[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #15880: [CI] unix cpu validation Timeout
ChaiBapchya commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521894929 Another one #15736 http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-15736/10/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15753: Numpy add numpy linalg norm and numpy min
haojin2 commented on a change in pull request #15753: Numpy add numpy linalg
norm and numpy min
URL: https://github.com/apache/incubator-mxnet/pull/15753#discussion_r314591015
##
File path: src/operator/numpy/np_broadcast_reduce_op_value.cc
##
@@ -77,9 +77,9 @@ NNVM_REGISTER_OP(_backward_np_sum)
inline bool IsIntType(const int dtype) {
return (dtype == mshadow::kUint8 ||
- dtype == mshadow::kInt32 ||
- dtype == mshadow::kInt8 ||
- dtype == mshadow::kInt64);
+ dtype == mshadow::kInt32 ||
Review comment:
Why making changes here? Did you apply some kind of linter to this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on issue #15703: Storage manager / memory usage regression in v1.5
apeforest commented on issue #15703: Storage manager / memory usage regression in v1.5 URL: https://github.com/apache/incubator-mxnet/issues/15703#issuecomment-521893896 Interestingly, I found that turning on the USE_INT64_TENSOR_SIZE flag (meaning using int64_t instead of int32_t as index_t type) will solve the OOM issue. Still rootcausing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] tingying2020 commented on a change in pull request #15838: [numpy] nonzero
tingying2020 commented on a change in pull request #15838: [numpy] nonzero URL: https://github.com/apache/incubator-mxnet/pull/15838#discussion_r314590819 ## File path: tests/python/gpu/test_operator_gpu.py ## @@ -2331,6 +2331,47 @@ def test_math(): for op in ops: run_math(op, shape, dtype, check_value=check_value) +@with_seed() +def test_np_nonzero(): +class TestNonzero(HybridBlock): +def __init__(self): +super(TestNonzero, self).__init__() + +def hybrid_forward(self, F, x): +return F.npx.nonzero(x) + +types = ['int32', 'int64', 'float64', 'float32', 'float16'] +for hybridize in [True, False]: +for shape in [(), + (1,), + (1, 1), + (1, 2, 3), + (1, 0), + (2, 0, 3) + ]: +for oneType in types: +if oneType is 'float16': +rtol=1e-2 +atol=1e-2 Review comment: Done, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] tingying2020 commented on a change in pull request #15838: [numpy] nonzero
tingying2020 commented on a change in pull request #15838: [numpy] nonzero URL: https://github.com/apache/incubator-mxnet/pull/15838#discussion_r314590829 ## File path: tests/python/gpu/test_operator_gpu.py ## @@ -2331,6 +2331,47 @@ def test_math(): for op in ops: run_math(op, shape, dtype, check_value=check_value) +@with_seed() +def test_np_nonzero(): +class TestNonzero(HybridBlock): +def __init__(self): +super(TestNonzero, self).__init__() + +def hybrid_forward(self, F, x): +return F.npx.nonzero(x) + +types = ['int32', 'int64', 'float64', 'float32', 'float16'] +for hybridize in [True, False]: +for shape in [(), + (1,), + (1, 1), + (1, 2, 3), + (1, 0), + (2, 0, 3) + ]: +for oneType in types: +if oneType is 'float16': +rtol=1e-2 +atol=1e-2 +else: +rtol=1e-3 +atol=1e-5 +test_nonzero = TestNonzero() +if hybridize: +test_nonzero.hybridize() +x = rand_ndarray(shape, dtype=oneType).as_np_ndarray() +np_out = np.nonzero(x.asnumpy()) +np_out = np.transpose(np_out) +mx_out = test_nonzero(x) +assert mx_out.shape == np_out.shape +assert_almost_equal(mx_out.asnumpy(), np_out, rtol, atol) + +# Test imperative once again +mx_out = mx.npx.nonzero(x) +np_out = np.nonzero(x.asnumpy()) +np_out = np.transpose(np_out) +assert_almost_equal(mx_out.asnumpy(), np_out, rtol, atol) + Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] tingying2020 commented on a change in pull request #15838: [numpy] nonzero
tingying2020 commented on a change in pull request #15838: [numpy] nonzero
URL: https://github.com/apache/incubator-mxnet/pull/15838#discussion_r314590614
##
File path: src/operator/numpy/np_nonzero_op.cu
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+/*!
+ * Copyright (c) 2018 by Contributors
+ * \file np_nonzero_op.cu
+*/
+
+#include "np_nonzero_op-inl.h"
+#include
+
+namespace mxnet {
+namespace op {
+
+struct PrefixSumInit{
Review comment:
Fixed, thank you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314590377 ## File path: include/mxnet/executor.h ## @@ -28,6 +28,8 @@ #include #include +#include Review comment: where is this needed and which warning did this fix? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589743 ## File path: src/operator/nn/mkldnn/mkldnn_concat.cc ## @@ -25,6 +25,7 @@ #if MXNET_USE_MKLDNN == 1 #include "mkldnn_concat-inl.h" +#include Review comment: where is this needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589730 ## File path: src/operator/nn/mkldnn/mkldnn_fully_connected.cc ## @@ -26,6 +26,7 @@ #if MXNET_USE_MKLDNN == 1 #include "mkldnn_fully_connected-inl.h" +#include Review comment: where is this needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589762 ## File path: src/operator/nn/cudnn/cudnn_algoreg-inl.h ## @@ -30,6 +30,7 @@ #include #include #include +#include Review comment: where is this needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings
URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589619
##
File path: src/common/exec_utils.h
##
@@ -49,85 +49,19 @@ namespace common {
indices are not recorded
* \return true if any source NDArray need to cast storage
*/
-inline bool SetupDefaultBlobsIn(const std::vector& src,
Review comment:
What warnings did this cause?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589474 ## File path: src/common/exec_utils.cc ## @@ -0,0 +1,519 @@ +/* Review comment: I failed to understand what warnings did this fix? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings
URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589551
##
File path: src/common/exec_utils.cc
##
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "exec_utils.h"
+
+namespace mxnet {
+namespace common {
+
+
+bool SetupDefaultBlobsIn(const std::vector& src,
+const std::vector *bufs,
+std::vector *blobs,
+std::vector *temp_src,
+std::vector *temp_dst,
+std::unordered_map
*idx_map) {
+ bool require_cast = false;
+ for (size_t i = 0; i < src.size(); i++) {
+auto& nd = src[i];
+bool is_default = nd.storage_type() == kDefaultStorage;
+#if MXNET_USE_MKLDNN == 1
+// We have to make sure it's default storage and default layout.
+is_default = nd.IsDefaultData();
+#endif
+if (!is_default) {
+ (*idx_map)[i] = temp_dst->size();
+ NDArray temp = bufs != nullptr ? bufs->at(i) : NDArray(nd.shape(),
nd.ctx(),
+ true, nd.dtype());
+#if MXNET_USE_MKLDNN == 1
+ CHECK(temp.IsDefaultData());
+#endif
+ temp_src->emplace_back(nd);
+ temp_dst->emplace_back(temp);
+ blobs->emplace_back(temp.data());
+ require_cast = true;
+} else {
+ blobs->push_back(nd.data());
+}
+ }
+ return require_cast;
+}
+
+bool SetupDefaultBlobsOut(const std::vector& src,
+ const std::vector *bufs,
+ std::vector *req,
+ std::vector *blobs,
+ std::vector *temp_src,
+ std::vector *temp_dst) {
+ bool require_cast = false;
+ for (size_t i = 0; i < src.size(); i++) {
+auto& nd = src[i];
+bool is_default = nd.storage_type() == kDefaultStorage;
+#if MXNET_USE_MKLDNN == 1
+if (req->at(i) == kWriteInplace && nd.IsMKLDNNData())
+ // If it's write inplace and the output array doesn't use the default
+ // layout, we'll generate a temporary output array below, which means
+ // the input array and the output array are no longer the same array.
+ // we should change the request type.
+ req->at(i) = kWriteTo;
+// We have to make sure it's default storage and default layout.
+is_default = nd.IsDefaultData();
+#endif
+if (!is_default) {
+#if MXNET_USE_MKLDNN == 1
+ NDArray temp;
+ if (bufs != nullptr) {
+temp = bufs->at(i);
+ } else if (kAddTo == req->at(i) && nd.IsMKLDNNData()) {
+temp = nd.Reorder2Default();
+ } else if (kAddTo == req->at(i)) {
+temp = nd;
+ } else {
+temp = NDArray(nd.shape(), nd.ctx(), true, nd.dtype());
+ }
+ CHECK(temp.IsDefaultData());
+#else
+ NDArray temp = bufs != nullptr ? bufs->at(i) : NDArray(nd.shape(),
nd.ctx(),
+ true, nd.dtype());
+#endif
+ temp_src->emplace_back(nd);
+ temp_dst->emplace_back(temp);
+ blobs->emplace_back(temp.data());
+ require_cast = true;
+} else {
+ blobs->push_back(nd.data());
+}
+ }
+ return require_cast;
+}
+
+
+void SetupDefaultBlobsInOut(const std::vector ,
+ const std::vector ,
+ const std::vector *in_bufs,
+ const std::vector *out_bufs,
+ std::vector *req,
+ std::vector *input_blobs,
+ std::vector *output_blobs,
+ std::vector *pre_temp_src,
+ std::vector *pre_temp_dst,
+ std::vector *post_temp_src,
+ std::vector *post_temp_dst,
+ std::unordered_map
*in_temp_idx_map,
+ const std::vector _idx) {
+ // populate input blobs
+ SetupDefaultBlobsIn(ndinputs, in_bufs, input_blobs, pre_temp_src,
[GitHub] [incubator-mxnet] zixuanweeei edited a comment on issue #15880: [CI] unix cpu validation Timeout
zixuanweeei edited a comment on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521890365 Some fixes from PR #15882 and PR #15922 (they have the same fixes on `test_shuffle`) has reduced the cost from more than 10 mins to no more than 2 mins. It cost ~41s on local test. And it seems that the fixes on `test_shuffle` doesn't alter the functionality of the test. It just drops the needless equal assertions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request #14940: Fix warnings
apeforest commented on a change in pull request #14940: Fix warnings
URL: https://github.com/apache/incubator-mxnet/pull/14940#discussion_r314589408
##
File path: src/common/exec_utils.cc
##
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "exec_utils.h"
+
+namespace mxnet {
+namespace common {
+
+
+bool SetupDefaultBlobsIn(const std::vector& src,
+const std::vector *bufs,
+std::vector *blobs,
+std::vector *temp_src,
+std::vector *temp_dst,
+std::unordered_map
*idx_map) {
+ bool require_cast = false;
+ for (size_t i = 0; i < src.size(); i++) {
+auto& nd = src[i];
+bool is_default = nd.storage_type() == kDefaultStorage;
+#if MXNET_USE_MKLDNN == 1
+// We have to make sure it's default storage and default layout.
+is_default = nd.IsDefaultData();
+#endif
+if (!is_default) {
+ (*idx_map)[i] = temp_dst->size();
+ NDArray temp = bufs != nullptr ? bufs->at(i) : NDArray(nd.shape(),
nd.ctx(),
+ true, nd.dtype());
+#if MXNET_USE_MKLDNN == 1
+ CHECK(temp.IsDefaultData());
+#endif
+ temp_src->emplace_back(nd);
+ temp_dst->emplace_back(temp);
+ blobs->emplace_back(temp.data());
+ require_cast = true;
+} else {
+ blobs->push_back(nd.data());
+}
+ }
+ return require_cast;
+}
+
+bool SetupDefaultBlobsOut(const std::vector& src,
+ const std::vector *bufs,
+ std::vector *req,
+ std::vector *blobs,
+ std::vector *temp_src,
+ std::vector *temp_dst) {
+ bool require_cast = false;
+ for (size_t i = 0; i < src.size(); i++) {
+auto& nd = src[i];
+bool is_default = nd.storage_type() == kDefaultStorage;
+#if MXNET_USE_MKLDNN == 1
+if (req->at(i) == kWriteInplace && nd.IsMKLDNNData())
+ // If it's write inplace and the output array doesn't use the default
+ // layout, we'll generate a temporary output array below, which means
+ // the input array and the output array are no longer the same array.
+ // we should change the request type.
+ req->at(i) = kWriteTo;
+// We have to make sure it's default storage and default layout.
+is_default = nd.IsDefaultData();
+#endif
+if (!is_default) {
+#if MXNET_USE_MKLDNN == 1
+ NDArray temp;
+ if (bufs != nullptr) {
+temp = bufs->at(i);
+ } else if (kAddTo == req->at(i) && nd.IsMKLDNNData()) {
+temp = nd.Reorder2Default();
+ } else if (kAddTo == req->at(i)) {
+temp = nd;
+ } else {
+temp = NDArray(nd.shape(), nd.ctx(), true, nd.dtype());
+ }
+ CHECK(temp.IsDefaultData());
+#else
+ NDArray temp = bufs != nullptr ? bufs->at(i) : NDArray(nd.shape(),
nd.ctx(),
+ true, nd.dtype());
+#endif
+ temp_src->emplace_back(nd);
+ temp_dst->emplace_back(temp);
+ blobs->emplace_back(temp.data());
+ require_cast = true;
+} else {
+ blobs->push_back(nd.data());
+}
+ }
+ return require_cast;
+}
+
+
+void SetupDefaultBlobsInOut(const std::vector ,
+ const std::vector ,
+ const std::vector *in_bufs,
+ const std::vector *out_bufs,
+ std::vector *req,
+ std::vector *input_blobs,
+ std::vector *output_blobs,
+ std::vector *pre_temp_src,
+ std::vector *pre_temp_dst,
+ std::vector *post_temp_src,
+ std::vector *post_temp_dst,
+ std::unordered_map
*in_temp_idx_map,
+ const std::vector _idx) {
+ // populate input blobs
+ SetupDefaultBlobsIn(ndinputs, in_bufs, input_blobs, pre_temp_src,
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15753: Numpy add numpy linalg norm and numpy min
haojin2 commented on a change in pull request #15753: Numpy add numpy linalg norm and numpy min URL: https://github.com/apache/incubator-mxnet/pull/15753#discussion_r314588556 ## File path: python/mxnet/symbol/numpy/_symbol.py ## @@ -1653,7 +1653,7 @@ def linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None, axis GPU. """ if isinstance(start, (list, _np.ndarray)) or \ -isinstance(stop, (list, _np.ndarray)): +isinstance(stop, (list, _np.ndarray)): Review comment: No need for extra indentation here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout
zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521890365 Some fixes from PR #15882 has reduced the cost from more than 10 mins to no more than 2 mins. It cost ~41s on local test. And it seems that the fixes on `test_shuffle` doesn't alter the functionality of the test. It just drops the needless equal assertions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15753: Numpy add numpy linalg norm and numpy min
haojin2 commented on a change in pull request #15753: Numpy add numpy linalg
norm and numpy min
URL: https://github.com/apache/incubator-mxnet/pull/15753#discussion_r314588331
##
File path: python/mxnet/ndarray/numpy/linalg.py
##
@@ -19,50 +19,184 @@
from __future__ import absolute_import
from . import _op as _mx_nd_np
+from . import _internal as _npi
+
__all__ = ['norm']
+# pylint: disable=line-too-long, too-many-return-statements,
too-many-branches, no-member, consider-using-in, no-else-return
def norm(x, ord=None, axis=None, keepdims=False):
-r"""Matrix or vector norm.
-
-This function can only support Frobenius norm for now.
-The Frobenius norm is given by [1]_:
-
-:math:`||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}`
+r"""Matrix or vector norm..
+This function is able to return one of eight different matrix norms,
+or one of an infinite number of vector norms (described below), depending
+on the value of the ``ord`` parameter.
Parameters
--
x : ndarray
-Input array.
-ord : {'fro'}, optional
-Order of the norm.
+Input array. If `axis` is None, `x` must be 1-D or 2-D.
+ord : {non-zero int, inf, -inf, 'fro', 'nuc'}, optional
+Order of the norm (see table under ``Notes``). inf means numpy's
+`inf` object.
axis : {int, 2-tuple of ints, None}, optional
If `axis` is an integer, it specifies the axis of `x` along which to
compute the vector norms. If `axis` is a 2-tuple, it specifies the
axes that hold 2-D matrices, and the matrix norms of these matrices
-are computed. If `axis` is None, the norm of the whole ndarray is
-returned.
-
+are computed. If `axis` is None then either a vector norm (when `x`
+is 1-D) or a matrix norm (when `x` is 2-D) is returned.
keepdims : bool, optional
If this is set to True, the axes which are normed over are left in the
result as dimensions with size one. With this option the result will
broadcast correctly against the original `x`.
Returns
---
-n : float or ndarray
+n : ndarray
Norm of the matrix or vector(s).
+Notes
+-
+For values of ``ord <= 0``, the result is, strictly speaking, not a
+mathematical 'norm', but it may still be useful for various numerical
+purposes.
+
+The following norms can be calculated:
+
+= ==
+ordnorm for matrices norm for vectors
+= ==
+None Frobenius norm2-norm
+'fro' Frobenius norm--
+'nuc' ----
+infmax(sum(abs(x), axis=1)) max(abs(x))
+-inf min(sum(abs(x), axis=1)) min(abs(x))
+0 --sum(x != 0)
+1 max(sum(abs(x), axis=0)) as below
+-1 min(sum(abs(x), axis=0)) as below
+2 --as below
+-2 --as below
+other --sum(abs(x)**ord)**(1./ord)
+= ==
+
+The Frobenius norm is given by [1]_:
+
+:math:`||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}`
+
+The nuclear norm is the sum of the singular values.
+
+When you want to operate norm for matrices,if you ord is (-1, 1, inf,
-inf),
+you must give you axis, it is not support default axis.
+
References
--
.. [1] G. H. Golub and C. F. Van Loan, *Matrix Computations*,
Baltimore, MD, Johns Hopkins University Press, 1985, pg. 15
+
+Examples
+
+>>> from mxnet import np
+>>> a = np.arange(9) - 4
+>>> a
+array([-4., -3., -2., -1., 0., 1., 2., 3., 4.])
+>>> b = a.reshape((3, 3))
+>>> b
+array([[-4., -3., -2.],
+ [-1., 0., 1.],
+ [ 2., 3., 4.]])
+
+>>> np.linalg.norm(a)
+array(7.745967)
+>>> np.linalg.norm(b)
+array(7.745967)
+>>> np.linalg.norm(b, 'fro')
+array(7.745967)
+>>> np.linalg.norm(a, 'inf')
+array(4.)
+>>> np.linalg.norm(b, 'inf', axis=(0, 1))
+array(9.)
+>>> np.linalg.norm(a, '-inf')
+array(0.)
+>>> np.linalg.norm(b, '-inf', axis=(0, 1))
+array(2.)
+
+>>> np.linalg.norm(a, 1)
+array(20.)
+>>> np.linalg.norm(b, 1, axis=(0, 1))
+array(7.)
+>>> np.linalg.norm(a, -1)
+array(0.)
+>>> np.linalg.norm(b, -1, axis=(0, 1))
+array(6.)
+>>> np.linalg.norm(a, 2)
+array(7.745967)
+
+>>> np.linalg.norm(a, -2)
+array(0.)
+>>> np.linalg.norm(a, 3)
+array(5.8480353)
+>>> np.linalg.norm(a, -3)
+array(0.)
+
+Using the `axis` argument to compute vector norms:
+
+>>> c = np.array([[ 1, 2, 3],
+...
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15753: Numpy add numpy linalg norm and numpy min
haojin2 commented on a change in pull request #15753: Numpy add numpy linalg
norm and numpy min
URL: https://github.com/apache/incubator-mxnet/pull/15753#discussion_r314588267
##
File path: python/mxnet/ndarray/numpy/linalg.py
##
@@ -19,50 +19,184 @@
from __future__ import absolute_import
from . import _op as _mx_nd_np
+from . import _internal as _npi
+
__all__ = ['norm']
+# pylint: disable=line-too-long, too-many-return-statements,
too-many-branches, no-member, consider-using-in, no-else-return
def norm(x, ord=None, axis=None, keepdims=False):
-r"""Matrix or vector norm.
-
-This function can only support Frobenius norm for now.
-The Frobenius norm is given by [1]_:
-
-:math:`||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}`
+r"""Matrix or vector norm..
+This function is able to return one of eight different matrix norms,
+or one of an infinite number of vector norms (described below), depending
+on the value of the ``ord`` parameter.
Parameters
--
x : ndarray
-Input array.
-ord : {'fro'}, optional
-Order of the norm.
+Input array. If `axis` is None, `x` must be 1-D or 2-D.
+ord : {non-zero int, inf, -inf, 'fro', 'nuc'}, optional
Review comment:
if something ('nuc') is not supported, firstly do not list it as an option.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] TaoLv commented on issue #15874: [v1.5.x] Handle fix_gamma in tensorrt subgraph conversion correctly
TaoLv commented on issue #15874: [v1.5.x] Handle fix_gamma in tensorrt subgraph conversion correctly URL: https://github.com/apache/incubator-mxnet/pull/15874#issuecomment-521889387 Thank you @KellenSunderland. Merging now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch v1.5.x updated: handle fix_gamma in tensorrt subgraph conversion correctly (#15645) (#15874)
This is an automated email from the ASF dual-hosted git repository.
taolv pushed a commit to branch v1.5.x
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git
The following commit(s) were added to refs/heads/v1.5.x by this push:
new 6a36152 handle fix_gamma in tensorrt subgraph conversion correctly
(#15645) (#15874)
6a36152 is described below
commit 6a361520c036d8dcb3144f89fb15917ce4f5039f
Author: Kellen Sunderland
AuthorDate: Thu Aug 15 22:30:21 2019 -0700
handle fix_gamma in tensorrt subgraph conversion correctly (#15645) (#15874)
---
src/operator/subgraph/tensorrt/nnvm_to_onnx-inl.h | 21 ++--
src/operator/subgraph/tensorrt/nnvm_to_onnx.cc| 29 +-
src/operator/subgraph/tensorrt/tensorrt.cc| 2 +-
tests/python/tensorrt/test_tensorrt_batchnorm.py | 65 +++
4 files changed, 109 insertions(+), 8 deletions(-)
diff --git a/src/operator/subgraph/tensorrt/nnvm_to_onnx-inl.h
b/src/operator/subgraph/tensorrt/nnvm_to_onnx-inl.h
index 4a88aee..f5bf8b7 100644
--- a/src/operator/subgraph/tensorrt/nnvm_to_onnx-inl.h
+++ b/src/operator/subgraph/tensorrt/nnvm_to_onnx-inl.h
@@ -33,6 +33,8 @@
#include
+#include
+#include
#include
namespace mxnet {
@@ -72,15 +74,12 @@ typedef void (*ConverterFunction)(NodeProto *node_proto,
const nnvm::IndexedGraph ,
const array_view
);
-
// Forward declarations
-void ConvertConvolution(
-NodeProto *node_proto,
+void ConvertConvolution(NodeProto *node_proto,
const NodeAttrs ,
const nnvm::IndexedGraph ,
const array_view );
-
void ConvertPooling(NodeProto *node_proto,
const NodeAttrs ,
const nnvm::IndexedGraph ,
@@ -142,7 +141,7 @@ void ConvertPad(NodeProto* node_proto,
const array_view );
std::string ConvertNnvmGraphToOnnx(const nnvm::Graph ,
-const std::unordered_map* const params_map);
+std::unordered_map* params_map);
static const std::unordered_map converter_map
= {
{"Activation", ConvertActivation},
@@ -160,6 +159,18 @@ static const std::unordered_map converter_map =
{"SoftmaxOutput", ConvertSoftmaxOutput}
};
+typedef void (*PreprocessFunction)(const NodeAttrs ,
+ const std::vector ,
+ std::unordered_map
*params_map);
+
+void PreprocessBatchNorm(const NodeAttrs ,
+ const std::vector ,
+ std::unordered_map *params_map);
+
+static const std::unordered_map
preprocess_map = {
+ {"BatchNorm", PreprocessBatchNorm}
+};
+
} // namespace nnvm_to_onnx
} // namespace op
} // namespace mxnet
diff --git a/src/operator/subgraph/tensorrt/nnvm_to_onnx.cc
b/src/operator/subgraph/tensorrt/nnvm_to_onnx.cc
index da89c2b..111995d 100644
--- a/src/operator/subgraph/tensorrt/nnvm_to_onnx.cc
+++ b/src/operator/subgraph/tensorrt/nnvm_to_onnx.cc
@@ -54,7 +54,7 @@ namespace nnvm_to_onnx {
std::string ConvertNnvmGraphToOnnx(
const nnvm::Graph& g,
-const std::unordered_map* const params_map) {
+std::unordered_map* params_map) {
static std::atomic_ulong subgraph_count = { 0 };
@@ -88,8 +88,21 @@ std::string ConvertNnvmGraphToOnnx(
auto placeholder_shapes = GetPlaceholderShapes(shape_inputs, ig);
auto placeholder_dtypes = GetPlaceholderDTypes(dtype_inputs, ig);
auto output_lookup = GetOutputLookup(ig);
- uint32_t current_input = 0;
+ for (uint32_t node_idx = 0; node_idx < ig.num_nodes(); ++node_idx) {
+ const IndexedGraph::Node& node = ig[node_idx];
+ const nnvm::Node* source = node.source;
+ // If this is a op
+ if (!source->is_variable()) {
+auto mightNeedPreprocessNode = preprocess_map.find(source->op()->name);
+// if this op is defined in preprocess_map
+if (mightNeedPreprocessNode != preprocess_map.end()) {
+ mightNeedPreprocessNode->second(source->attrs, source->inputs,
params_map);
+}
+ }
+ }
+
+ uint32_t current_input = 0;
// Can't do a foreach over IndexedGraph since it doesn't implement begin(),
etc.
for (uint32_t node_idx = 0; node_idx < ig.num_nodes(); ++node_idx) {
const IndexedGraph::Node& node = ig[node_idx];
@@ -630,6 +643,18 @@ void ConvertDropout(NodeProto* node_proto, const
NodeAttrs& attrs,
node_proto->set_op_type("Dropout");
}
+void PreprocessBatchNorm(const NodeAttrs ,
+ const std::vector ,
+ std::unordered_map *params_map)
{
+ const auto& param = nnvm::get(attrs.parsed);
+ if (param.fix_gamma) {
+// if mxnet is specify fix_gamma, we will need to preprocess the params map
+// to convert the gamma associate with this batch norm layer to 1.
+std::string gammaNodeName = inputs[batchnorm::kGamma].node->attrs.name;
+
[GitHub] [incubator-mxnet] TaoLv merged pull request #15874: [v1.5.x] Handle fix_gamma in tensorrt subgraph conversion correctly
TaoLv merged pull request #15874: [v1.5.x] Handle fix_gamma in tensorrt subgraph conversion correctly URL: https://github.com/apache/incubator-mxnet/pull/15874 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] tingying2020 commented on a change in pull request #15819: [Numpy]flip
tingying2020 commented on a change in pull request #15819: [Numpy]flip URL: https://github.com/apache/incubator-mxnet/pull/15819#discussion_r314587498 ## File path: tests/python/gpu/test_operator_gpu.py ## @@ -2331,6 +2331,72 @@ def test_math(): for op in ops: run_math(op, shape, dtype, check_value=check_value) + +@with_seed() +@use_np +def test_np_flip(): Review comment: Done, thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] aaronmarkham commented on issue #8773: undefined reference to `gotoblas' Makefile:406: recipe for target 'bin/im2rec' failed
aaronmarkham commented on issue #8773: undefined reference to `gotoblas' Makefile:406: recipe for target 'bin/im2rec' failed URL: https://github.com/apache/incubator-mxnet/issues/8773#issuecomment-521888720 I keep running into this issue as well. A clean clone with recursive init of the submodules seems to get around it, but this isn't a real fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582942
##
File path: src/operator/numpy/np_matrix_op.cc
##
@@ -135,17 +136,165 @@ bool NumpyReshapeInferShape(const mxnet::TShape& src,
mxnet::TShape* dst) {
}
}
+bool NumpyXReshapeInferShape(const mxnet::TShape& src,
+ const mxnet::Tuple& target,
+ mxnet::TShape* output) {
+ bool target_shape_is_known = true;
+ dim_t target_size = 1;
+ for (int i = 0; i < target.ndim(); ++i) {
+if (target[i] < 0) {
+ target_shape_is_known = false;
+ target_size = -1;
+ break;
+} else {
+ target_size *= target[i];
+}
+ }
+ if (shape_is_known(src) && target_shape_is_known) {
+CHECK_EQ(src.Size(), target_size) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+*output = TShape(target.begin(), target.end());
+return true;
+ } else if (!shape_is_known(src) || target.ndim() == -1) {
+return false;
+ } else {
+int unknown_axis = -1;
+dim_t known_dim_size_prod = 1;
+std::vector output_shape_vector;
+int src_inx = 0;
+for (int i = 0; i < target.ndim(); ++i) {
+ dim_t proposed_dim = target[i];
+ CHECK(proposed_dim >= -6)
+<< "Dimension size must be greater than -6, received " << proposed_dim;
+ if (proposed_dim == -1) {
+// infer the known dimension
+CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+unknown_axis = output_shape_vector.size();
+output_shape_vector.push_back(1);
+src_inx++;
+ } else if (proposed_dim == -2) {
+// copy the dimension from src to output
+CHECK_LT(src_inx, src.ndim())
+ << "Unmatching dimension of proposed new shape";
+known_dim_size_prod *= src[src_inx];
+output_shape_vector.push_back(src[src_inx++]);
+ } else if (proposed_dim == -3) {
+// skip the source dimension if and only if it is one
+CHECK_EQ(src[src_inx], 1)
+ <<"-3 index should only be used to skip dimision size 1";
+src_inx++;
+ } else if (proposed_dim == -4) {
+// copy all remaining dims from source
+while (src_inx < src.ndim()) {
+ known_dim_size_prod *= src[src_inx];
+ const int dn = src[src_inx++];
+ output_shape_vector.push_back(dn);
+}
+ } else if (proposed_dim == -5) {
+// merge two dims from source
+CHECK_LT(src_inx, src.ndim()-1)
+ <<"Not enough dimensions left for the product";
+const int d1 = src[src_inx++];
+const int d2 = src[src_inx++];
+if (!mxnet::dim_size_is_known(d1) || !mxnet::dim_size_is_known(d2)) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+ output_shape_vector.push_back(-1);
+} else {
+ known_dim_size_prod *= d1*d2;
+ output_shape_vector.push_back(d1 * d2);
+}
+ } else if (proposed_dim == -6) {
+// split the source dim s into two dims
+// read the left dim and then the right dim (either can be -1)
+CHECK_LT(i + 2, target.ndim());
+CHECK_LT(src_inx, src.ndim());
+const int d0 = src[src_inx++];
+dim_t d1 = target[++i];
+dim_t d2 = target[++i];
+CHECK(d1 != -1 || d2 != -1) << "Split dims cannot both be -1.";
+if (d1 == -1 && d0 >= 0) d1 = d0 / d2; // d0 must be known to do this
+if (d2 == -1 && d0 >= 0) d2 = d0 / d1; // d0 must be known to do this
+CHECK(d1 * d2 == static_cast(d0) || static_cast(d0) ==
dim_t(-1))
+ <<"Split dims " << d1 << ", " << d2 << " do not divide original dim
" << d0;
+if (d1 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+} else if (d2 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size() + 1;
+}
+known_dim_size_prod *= d0 == -1 ? 1 : d0;
+output_shape_vector.push_back(d1);
+output_shape_vector.push_back(d2);
+ } else {
+// greater than 0, new shape
+known_dim_size_prod *= proposed_dim;
+output_shape_vector.push_back(proposed_dim);
+src_inx++;
+ }
+}
+
+if (unknown_axis > -1) {
+ // if the input in zero size tensor, the output must be of known shape
of zero size
+ CHECK_NE(known_dim_size_prod, 0) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+ CHECK(src.Size() % known_dim_size_prod == 0)
+
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582869
##
File path: src/operator/numpy/np_matrix_op.cc
##
@@ -135,17 +136,165 @@ bool NumpyReshapeInferShape(const mxnet::TShape& src,
mxnet::TShape* dst) {
}
}
+bool NumpyXReshapeInferShape(const mxnet::TShape& src,
+ const mxnet::Tuple& target,
+ mxnet::TShape* output) {
+ bool target_shape_is_known = true;
+ dim_t target_size = 1;
+ for (int i = 0; i < target.ndim(); ++i) {
+if (target[i] < 0) {
+ target_shape_is_known = false;
+ target_size = -1;
+ break;
+} else {
+ target_size *= target[i];
+}
+ }
+ if (shape_is_known(src) && target_shape_is_known) {
+CHECK_EQ(src.Size(), target_size) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+*output = TShape(target.begin(), target.end());
+return true;
+ } else if (!shape_is_known(src) || target.ndim() == -1) {
+return false;
+ } else {
+int unknown_axis = -1;
+dim_t known_dim_size_prod = 1;
+std::vector output_shape_vector;
+int src_inx = 0;
+for (int i = 0; i < target.ndim(); ++i) {
+ dim_t proposed_dim = target[i];
+ CHECK(proposed_dim >= -6)
+<< "Dimension size must be greater than -6, received " << proposed_dim;
+ if (proposed_dim == -1) {
+// infer the known dimension
+CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+unknown_axis = output_shape_vector.size();
+output_shape_vector.push_back(1);
+src_inx++;
+ } else if (proposed_dim == -2) {
+// copy the dimension from src to output
+CHECK_LT(src_inx, src.ndim())
+ << "Unmatching dimension of proposed new shape";
+known_dim_size_prod *= src[src_inx];
+output_shape_vector.push_back(src[src_inx++]);
+ } else if (proposed_dim == -3) {
+// skip the source dimension if and only if it is one
+CHECK_EQ(src[src_inx], 1)
+ <<"-3 index should only be used to skip dimision size 1";
+src_inx++;
+ } else if (proposed_dim == -4) {
+// copy all remaining dims from source
+while (src_inx < src.ndim()) {
+ known_dim_size_prod *= src[src_inx];
+ const int dn = src[src_inx++];
+ output_shape_vector.push_back(dn);
+}
+ } else if (proposed_dim == -5) {
+// merge two dims from source
+CHECK_LT(src_inx, src.ndim()-1)
+ <<"Not enough dimensions left for the product";
+const int d1 = src[src_inx++];
+const int d2 = src[src_inx++];
+if (!mxnet::dim_size_is_known(d1) || !mxnet::dim_size_is_known(d2)) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+ output_shape_vector.push_back(-1);
+} else {
+ known_dim_size_prod *= d1*d2;
+ output_shape_vector.push_back(d1 * d2);
+}
+ } else if (proposed_dim == -6) {
+// split the source dim s into two dims
+// read the left dim and then the right dim (either can be -1)
+CHECK_LT(i + 2, target.ndim());
+CHECK_LT(src_inx, src.ndim());
+const int d0 = src[src_inx++];
+dim_t d1 = target[++i];
+dim_t d2 = target[++i];
+CHECK(d1 != -1 || d2 != -1) << "Split dims cannot both be -1.";
+if (d1 == -1 && d0 >= 0) d1 = d0 / d2; // d0 must be known to do this
+if (d2 == -1 && d0 >= 0) d2 = d0 / d1; // d0 must be known to do this
+CHECK(d1 * d2 == static_cast(d0) || static_cast(d0) ==
dim_t(-1))
+ <<"Split dims " << d1 << ", " << d2 << " do not divide original dim
" << d0;
+if (d1 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+} else if (d2 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size() + 1;
+}
+known_dim_size_prod *= d0 == -1 ? 1 : d0;
+output_shape_vector.push_back(d1);
+output_shape_vector.push_back(d2);
+ } else {
+// greater than 0, new shape
+known_dim_size_prod *= proposed_dim;
+output_shape_vector.push_back(proposed_dim);
+src_inx++;
+ }
+}
+
+if (unknown_axis > -1) {
+ // if the input in zero size tensor, the output must be of known shape
of zero size
+ CHECK_NE(known_dim_size_prod, 0) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
Review comment:
Alignment:
```c++
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582829
##
File path: src/operator/numpy/np_matrix_op.cc
##
@@ -135,17 +136,165 @@ bool NumpyReshapeInferShape(const mxnet::TShape& src,
mxnet::TShape* dst) {
}
}
+bool NumpyXReshapeInferShape(const mxnet::TShape& src,
+ const mxnet::Tuple& target,
+ mxnet::TShape* output) {
+ bool target_shape_is_known = true;
+ dim_t target_size = 1;
+ for (int i = 0; i < target.ndim(); ++i) {
+if (target[i] < 0) {
+ target_shape_is_known = false;
+ target_size = -1;
+ break;
+} else {
+ target_size *= target[i];
+}
+ }
+ if (shape_is_known(src) && target_shape_is_known) {
+CHECK_EQ(src.Size(), target_size) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+*output = TShape(target.begin(), target.end());
+return true;
+ } else if (!shape_is_known(src) || target.ndim() == -1) {
+return false;
+ } else {
+int unknown_axis = -1;
+dim_t known_dim_size_prod = 1;
+std::vector output_shape_vector;
+int src_inx = 0;
+for (int i = 0; i < target.ndim(); ++i) {
+ dim_t proposed_dim = target[i];
+ CHECK(proposed_dim >= -6)
+<< "Dimension size must be greater than -6, received " << proposed_dim;
+ if (proposed_dim == -1) {
+// infer the known dimension
+CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+unknown_axis = output_shape_vector.size();
+output_shape_vector.push_back(1);
+src_inx++;
+ } else if (proposed_dim == -2) {
+// copy the dimension from src to output
+CHECK_LT(src_inx, src.ndim())
+ << "Unmatching dimension of proposed new shape";
+known_dim_size_prod *= src[src_inx];
+output_shape_vector.push_back(src[src_inx++]);
+ } else if (proposed_dim == -3) {
+// skip the source dimension if and only if it is one
+CHECK_EQ(src[src_inx], 1)
+ <<"-3 index should only be used to skip dimision size 1";
+src_inx++;
+ } else if (proposed_dim == -4) {
+// copy all remaining dims from source
+while (src_inx < src.ndim()) {
+ known_dim_size_prod *= src[src_inx];
+ const int dn = src[src_inx++];
+ output_shape_vector.push_back(dn);
+}
+ } else if (proposed_dim == -5) {
+// merge two dims from source
+CHECK_LT(src_inx, src.ndim()-1)
+ <<"Not enough dimensions left for the product";
+const int d1 = src[src_inx++];
+const int d2 = src[src_inx++];
+if (!mxnet::dim_size_is_known(d1) || !mxnet::dim_size_is_known(d2)) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+ output_shape_vector.push_back(-1);
+} else {
+ known_dim_size_prod *= d1*d2;
+ output_shape_vector.push_back(d1 * d2);
+}
+ } else if (proposed_dim == -6) {
+// split the source dim s into two dims
+// read the left dim and then the right dim (either can be -1)
+CHECK_LT(i + 2, target.ndim());
+CHECK_LT(src_inx, src.ndim());
+const int d0 = src[src_inx++];
+dim_t d1 = target[++i];
+dim_t d2 = target[++i];
+CHECK(d1 != -1 || d2 != -1) << "Split dims cannot both be -1.";
+if (d1 == -1 && d0 >= 0) d1 = d0 / d2; // d0 must be known to do this
+if (d2 == -1 && d0 >= 0) d2 = d0 / d1; // d0 must be known to do this
+CHECK(d1 * d2 == static_cast(d0) || static_cast(d0) ==
dim_t(-1))
+ <<"Split dims " << d1 << ", " << d2 << " do not divide original dim
" << d0;
+if (d1 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
Review comment:
Same for all applicable places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582898
##
File path: src/operator/numpy/np_matrix_op.cc
##
@@ -135,17 +136,165 @@ bool NumpyReshapeInferShape(const mxnet::TShape& src,
mxnet::TShape* dst) {
}
}
+bool NumpyXReshapeInferShape(const mxnet::TShape& src,
+ const mxnet::Tuple& target,
+ mxnet::TShape* output) {
+ bool target_shape_is_known = true;
+ dim_t target_size = 1;
+ for (int i = 0; i < target.ndim(); ++i) {
+if (target[i] < 0) {
+ target_shape_is_known = false;
+ target_size = -1;
+ break;
+} else {
+ target_size *= target[i];
+}
+ }
+ if (shape_is_known(src) && target_shape_is_known) {
+CHECK_EQ(src.Size(), target_size) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+*output = TShape(target.begin(), target.end());
+return true;
+ } else if (!shape_is_known(src) || target.ndim() == -1) {
+return false;
+ } else {
+int unknown_axis = -1;
+dim_t known_dim_size_prod = 1;
+std::vector output_shape_vector;
+int src_inx = 0;
+for (int i = 0; i < target.ndim(); ++i) {
+ dim_t proposed_dim = target[i];
+ CHECK(proposed_dim >= -6)
+<< "Dimension size must be greater than -6, received " << proposed_dim;
+ if (proposed_dim == -1) {
+// infer the known dimension
+CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+unknown_axis = output_shape_vector.size();
+output_shape_vector.push_back(1);
+src_inx++;
+ } else if (proposed_dim == -2) {
+// copy the dimension from src to output
+CHECK_LT(src_inx, src.ndim())
+ << "Unmatching dimension of proposed new shape";
+known_dim_size_prod *= src[src_inx];
+output_shape_vector.push_back(src[src_inx++]);
+ } else if (proposed_dim == -3) {
+// skip the source dimension if and only if it is one
+CHECK_EQ(src[src_inx], 1)
+ <<"-3 index should only be used to skip dimision size 1";
+src_inx++;
+ } else if (proposed_dim == -4) {
+// copy all remaining dims from source
+while (src_inx < src.ndim()) {
+ known_dim_size_prod *= src[src_inx];
+ const int dn = src[src_inx++];
+ output_shape_vector.push_back(dn);
+}
+ } else if (proposed_dim == -5) {
+// merge two dims from source
+CHECK_LT(src_inx, src.ndim()-1)
+ <<"Not enough dimensions left for the product";
+const int d1 = src[src_inx++];
+const int d2 = src[src_inx++];
+if (!mxnet::dim_size_is_known(d1) || !mxnet::dim_size_is_known(d2)) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+ output_shape_vector.push_back(-1);
+} else {
+ known_dim_size_prod *= d1*d2;
+ output_shape_vector.push_back(d1 * d2);
+}
+ } else if (proposed_dim == -6) {
+// split the source dim s into two dims
+// read the left dim and then the right dim (either can be -1)
+CHECK_LT(i + 2, target.ndim());
+CHECK_LT(src_inx, src.ndim());
+const int d0 = src[src_inx++];
+dim_t d1 = target[++i];
+dim_t d2 = target[++i];
+CHECK(d1 != -1 || d2 != -1) << "Split dims cannot both be -1.";
+if (d1 == -1 && d0 >= 0) d1 = d0 / d2; // d0 must be known to do this
+if (d2 == -1 && d0 >= 0) d2 = d0 / d1; // d0 must be known to do this
+CHECK(d1 * d2 == static_cast(d0) || static_cast(d0) ==
dim_t(-1))
+ <<"Split dims " << d1 << ", " << d2 << " do not divide original dim
" << d0;
+if (d1 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+} else if (d2 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size() + 1;
+}
+known_dim_size_prod *= d0 == -1 ? 1 : d0;
+output_shape_vector.push_back(d1);
+output_shape_vector.push_back(d2);
+ } else {
+// greater than 0, new shape
+known_dim_size_prod *= proposed_dim;
+output_shape_vector.push_back(proposed_dim);
+src_inx++;
+ }
+}
+
+if (unknown_axis > -1) {
+ // if the input in zero size tensor, the output must be of known shape
of zero size
+ CHECK_NE(known_dim_size_prod, 0) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+ CHECK(src.Size() % known_dim_size_prod == 0)
+
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582779
##
File path: src/operator/numpy/np_matrix_op.cc
##
@@ -135,17 +136,165 @@ bool NumpyReshapeInferShape(const mxnet::TShape& src,
mxnet::TShape* dst) {
}
}
+bool NumpyXReshapeInferShape(const mxnet::TShape& src,
+ const mxnet::Tuple& target,
+ mxnet::TShape* output) {
+ bool target_shape_is_known = true;
+ dim_t target_size = 1;
+ for (int i = 0; i < target.ndim(); ++i) {
+if (target[i] < 0) {
+ target_shape_is_known = false;
+ target_size = -1;
+ break;
+} else {
+ target_size *= target[i];
+}
+ }
+ if (shape_is_known(src) && target_shape_is_known) {
+CHECK_EQ(src.Size(), target_size) << "Cannot reshape array of size "
+ << src.Size() << " into shape " <<
target;
+*output = TShape(target.begin(), target.end());
+return true;
+ } else if (!shape_is_known(src) || target.ndim() == -1) {
+return false;
+ } else {
+int unknown_axis = -1;
+dim_t known_dim_size_prod = 1;
+std::vector output_shape_vector;
+int src_inx = 0;
+for (int i = 0; i < target.ndim(); ++i) {
+ dim_t proposed_dim = target[i];
+ CHECK(proposed_dim >= -6)
+<< "Dimension size must be greater than -6, received " << proposed_dim;
+ if (proposed_dim == -1) {
+// infer the known dimension
+CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+unknown_axis = output_shape_vector.size();
+output_shape_vector.push_back(1);
+src_inx++;
+ } else if (proposed_dim == -2) {
+// copy the dimension from src to output
+CHECK_LT(src_inx, src.ndim())
+ << "Unmatching dimension of proposed new shape";
+known_dim_size_prod *= src[src_inx];
+output_shape_vector.push_back(src[src_inx++]);
+ } else if (proposed_dim == -3) {
+// skip the source dimension if and only if it is one
+CHECK_EQ(src[src_inx], 1)
+ <<"-3 index should only be used to skip dimision size 1";
+src_inx++;
+ } else if (proposed_dim == -4) {
+// copy all remaining dims from source
+while (src_inx < src.ndim()) {
+ known_dim_size_prod *= src[src_inx];
+ const int dn = src[src_inx++];
+ output_shape_vector.push_back(dn);
+}
+ } else if (proposed_dim == -5) {
+// merge two dims from source
+CHECK_LT(src_inx, src.ndim()-1)
+ <<"Not enough dimensions left for the product";
+const int d1 = src[src_inx++];
+const int d2 = src[src_inx++];
+if (!mxnet::dim_size_is_known(d1) || !mxnet::dim_size_is_known(d2)) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
+ unknown_axis = output_shape_vector.size();
+ output_shape_vector.push_back(-1);
+} else {
+ known_dim_size_prod *= d1*d2;
+ output_shape_vector.push_back(d1 * d2);
+}
+ } else if (proposed_dim == -6) {
+// split the source dim s into two dims
+// read the left dim and then the right dim (either can be -1)
+CHECK_LT(i + 2, target.ndim());
+CHECK_LT(src_inx, src.ndim());
+const int d0 = src[src_inx++];
+dim_t d1 = target[++i];
+dim_t d2 = target[++i];
+CHECK(d1 != -1 || d2 != -1) << "Split dims cannot both be -1.";
+if (d1 == -1 && d0 >= 0) d1 = d0 / d2; // d0 must be known to do this
+if (d2 == -1 && d0 >= 0) d2 = d0 / d1; // d0 must be known to do this
+CHECK(d1 * d2 == static_cast(d0) || static_cast(d0) ==
dim_t(-1))
+ <<"Split dims " << d1 << ", " << d2 << " do not divide original dim
" << d0;
+if (d1 == -1) {
+ CHECK_LT(unknown_axis, 0)
+ << "One and only one dim can be inferred";
Review comment:
indent here:
```c++
CHECK_LT(unknown_axis, 0)
<< "One and only one dim can be inferred";
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582546
##
File path: python/mxnet/symbol/numpy/_symbol.py
##
@@ -39,25 +46,90 @@ def _num_outputs(sym):
@set_module('mxnet.symbol.numpy')
class _Symbol(Symbol):
-def __getitem__(self, key):
-num_outputs = _num_outputs(self)
-if num_outputs == 1:
-raise NotImplementedError
-if not isinstance(key, int):
+def __init__(self, handle):
+super(_Symbol, self).__init__(handle)
+self._output_is_list = False
+
+def __getitem__(self, key): # pylint: disable =
too-many-return-statements, inconsistent-return-statements
+num_outputs = len(self)
+# print("Num of outputs is ", num_outputs)
+if num_outputs == 1: # pylint: disable = too-many-nested-blocks
+# If number of output is one and is not a list, perform ndarray
basic slicing
+if not self._output_is_list:
+if isinstance(key, integer_types):
+sliced = _npi.slice(self, key, key+1)
+return _npi.reshape(sliced, (-3, -4))
+elif isinstance(key, py_slice):
+if key.step is None or key.step != 0:
+start = [None] if key.start is None else key.start
+stop = [None] if key.stop is None else key.stop
+return _npi.slice(self, start, stop, key.step)
+else:
+raise ValueError("slice step cannot be zero")
+elif isinstance(key, list):
+raise NotImplementedError
+elif isinstance(key, tuple):
+begin = []
+end = []
+step = []
+new_shape = ()
+for index in key:
+if isinstance(index, py_slice):
+if index.step is not None and index.step == 0:
+raise ValueError("slice step cannot be zero")
+begin.append(index.start)
+end.append(index.stop)
+step.append(index.step)
+new_shape += (-2,)
+elif isinstance(index, integer_types):
+begin.append(index)
+end.append(index+1)
+step.append(1)
+new_shape += (-3,)
+new_shape += (-4,)
+sliced = _npi.slice(self, begin, end, step)
+return _npi.reshape(sliced, new_shape)
+# perform trivial list slicing on length one list represented by
flag
+else:
+if isinstance(key, integer_types):
+if key in [-1, 0]:
+self._output_is_list = False
+return self
+else:
+raise IndexError
+elif isinstance(key, py_slice):
+if (key.start is None or key.start <= 0) and (key.stop is
None or key.stop > 0):
+return self
+else:
+raise ValueError
+else:
+raise IndexError
+# list slicing on several nodes of outputs
+elif num_outputs > 1:
+if isinstance(key, py_slice):
+start = 0 if key.start is None else key.start
+stop = num_outputs if key.stop is None else key.stop
+step = 1 if key.step is None else key.step
+return Group([self[i] for i in range(start, stop, step)],
_Symbol)
+elif isinstance(key, integer_types):
+if key >= num_outputs:
+# Important, python determines the end by this exception
+raise IndexError
+handle = SymbolHandle()
+check_call(_LIB.MXSymbolGetOutput(
+self.handle, mx_uint(key), ctypes.byref(handle)))
+return _Symbol(handle=handle)
+else:
+raise NotImplementedError
+else:
raise NotImplementedError
-if key >= num_outputs:
-# Important, python determines the end by this exception
-raise IndexError
-handle = SymbolHandle()
-check_call(_LIB.MXSymbolGetOutput(
-self.handle, mx_uint(key), ctypes.byref(handle)))
-return _Symbol(handle=handle)
+
Review comment:
get rid of the extra blank line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
[GitHub] [incubator-mxnet] haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in symbolic interface
haojin2 commented on a change in pull request #15905: [Numpy] Basic indexing in
symbolic interface
URL: https://github.com/apache/incubator-mxnet/pull/15905#discussion_r314582455
##
File path: python/mxnet/symbol/numpy/_symbol.py
##
@@ -39,25 +46,90 @@ def _num_outputs(sym):
@set_module('mxnet.symbol.numpy')
class _Symbol(Symbol):
-def __getitem__(self, key):
-num_outputs = _num_outputs(self)
-if num_outputs == 1:
-raise NotImplementedError
-if not isinstance(key, int):
+def __init__(self, handle):
+super(_Symbol, self).__init__(handle)
+self._output_is_list = False
+
+def __getitem__(self, key): # pylint: disable =
too-many-return-statements, inconsistent-return-statements
+num_outputs = len(self)
+# print("Num of outputs is ", num_outputs)
Review comment:
Get rid of dead code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #15918: Trainer.save_state does not work if update_on_kvstore=False
ptrendx commented on issue #15918: Trainer.save_state does not work if update_on_kvstore=False URL: https://github.com/apache/incubator-mxnet/issues/15918#issuecomment-521876652 Shouldn't those states be exactly the same for all contexts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] yzhliu commented on a change in pull request #15889: [DO NOT MERGE] enable tvm_op for ci
yzhliu commented on a change in pull request #15889: [DO NOT MERGE] enable
tvm_op for ci
URL: https://github.com/apache/incubator-mxnet/pull/15889#discussion_r31457
##
File path: tests/python/unittest/test_tvm_op.py
##
@@ -24,14 +24,13 @@
@with_seed()
def test_tvm_broadcast_add():
-if _features.is_enabled("TVM_OP"):
Review comment:
yes I will, thanks for reminding. could you guide me where to pip install
package in docker? ci failed with "ImportError: No module named 'decorator'"
during compiling
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] yzhliu commented on a change in pull request #15889: [DO NOT MERGE] enable tvm_op for ci
yzhliu commented on a change in pull request #15889: [DO NOT MERGE] enable
tvm_op for ci
URL: https://github.com/apache/incubator-mxnet/pull/15889#discussion_r31457
##
File path: tests/python/unittest/test_tvm_op.py
##
@@ -24,14 +24,13 @@
@with_seed()
def test_tvm_broadcast_add():
-if _features.is_enabled("TVM_OP"):
Review comment:
I will, thanks for reminding. could you guide me where to pip install
package in docker? ci failed with "ImportError: No module named 'decorator'"
during compiling
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] mxnet-label-bot commented on issue #15927: sym.Variable input init need init.dumps(), but NDarray is not JSON serializable
mxnet-label-bot commented on issue #15927: sym.Variable input init need init.dumps(), but NDarray is not JSON serializable URL: https://github.com/apache/incubator-mxnet/issues/15927#issuecomment-521873240 Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Bug, Feature This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] david-di opened a new issue #15927: sym.Variable input init need init.dumps(), but NDarray is not JSON serializable
david-di opened a new issue #15927: sym.Variable input init need init.dumps(), but NDarray is not JSON serializable URL: https://github.com/apache/incubator-mxnet/issues/15927 ## Description mxnet.symbol.Variable(name, attr=None, shape=None, lr_mult=None, wd_mult=None, dtype=None, init=None, stype=None, **kwargs) The input parameter init, I used mxnet.initializer.Constant(value). The introduction of Constant class says "Initializes the weights to a given value. The value passed in can be a scalar or a NDarray that matches the shape of the parameter to be set", so I use a NDarray as "value" parameter. Now, accident happened.. In class Variable, if init is not an instance of string, init.dumps() will be called. "dumps" method is implemented in class mxnet.initializer.Initializer, the superclass of Constant. ``` def dumps(self): return json.dumps([self.__class__.__name__.lower(), self._kwargs]) ``` NDarray is not JSON serializable... Crash... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #15922: Refactor for windows CI 'out of heap space' errors
ptrendx commented on issue #15922: Refactor for windows CI 'out of heap space' errors URL: https://github.com/apache/incubator-mxnet/pull/15922#issuecomment-521870531 As mentioned to Dick offline, I do not like this PR to be seen as an alternative to the #15882 PR. I believe we should do both of them - refactor the biggest files to speed up the compilation, but also guard against problems with single operators being too big. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch ib/jl-mxnet-root-env updated (36076c4 -> c0ab006)
This is an automated email from the ASF dual-hosted git repository. iblis pushed a change to branch ib/jl-mxnet-root-env in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from 36076c4 Merge branch 'master' into ib/jl-mxnet-root-env add 67acbe5 [DOC] Fix many arguments in the doc: reshape_like, arange_like, shape_array (#15752) add 8c2bef5 Add Gather_nd Scatter_nd to NDArray API category doc (#15689) add 51c07e5 [Dependency Update] Dependency update doc (#15045) add ce62873 Add quantization support for GluonCV (#15754) add 7186123 Revert "Dynamic Library Loading Support" (#15755) add aadef2d Fix flaky test test_global_metric (#15756) add 45db8ea Add matrix determinant operator in linalg (#15007) add 07eb482 fix tvm cmake (#15781) add a2b11ae Fix PR #15489 (Dynamic Library Loading Support) (#15760) add be49b3b Numpy-compatible Infra (#15581) add 79d8d86 Fix the bug of `MXEnginePushAsyncND` and `MXEnginePushSyncND` (#15751) add b6972bb add int8 bn mkldnn implementation and test (#15664) add b77f524 [MXNET-1206] Support NDArray indexing with None and Ellipsis (#13143) add a3babc4 numpy-compatible sum (#15810) add 19916af fix license issues (#15806) add bfd3bb8 Refactor LibraryInitializer so it's thread safe. Fixes random sporadical concurrency crashes. (#15762) add 0eb213d Fix backward_clip num inputs and type of clip params (#15688) add 44a7fca [Numpy] Numpy compatible slicing (#15798) add 614cba3 Making Features as a singleton for improved caching (#15835) add 57927a9 Clojure package remove source images (#15828) add c3f5eea Numpy Tensordot and Dot Operator (#15820) add bd32de4 fixing problem with existing Singleton Caching (#15868) add 795990b numpy linspace (#15852) add 05f3ae1 Large Index Support for Slice (#15593) add 67daae7 tvm infra for op attrs (#15854) add c81535c cuDNN support cleanup (#15812) add 39bf4e0 Port several np ops to master (#15867) add 11ce2a2 Add large tensor support binary arithmetic (#15785) add 24a5cf0 Allow operators with multiple outputs in get_atomic_symbol (#15740) add f32b58e numpy-compatible split upstream (#15841) add b914d0a Large tensor support for random ops (#15783) add 843c3ab Add Large Tensor Support for Sequence, NN Ops (#15807) add 3d8035b Numpy-compatible concatenate upstream (#15894) add 78733f0 Numpy-compatible concatenate upstream (#15894) add b98f297 Numpy-compatible concatenate upstream (#15894) add 9dbfc2d Numpy-compatible concatenate upstream (#15894) add 40593c6 Fix ConcatType backward type inference (#15829) add f586c07 [OpPerf] Profiler flag for Python, Cpp (#15881) add acc7865 added check for empty params file and unknown param (not arg/aux) (#15917) add c8f0b21 Remove Scala package test on build (#15915) add c0ab006 Merge branch 'master' into ib/jl-mxnet-root-env No new revisions were added by this update. Summary of changes: CMakeLists.txt | 19 +- LICENSE| 355 +++- Makefile |5 +- benchmark/opperf/nd_operations/array_rearrange.py |4 +- benchmark/opperf/nd_operations/binary_operators.py |8 +- benchmark/opperf/nd_operations/gemm_operators.py |6 +- .../nd_operations/nn_activation_operators.py |5 +- .../opperf/nd_operations/nn_basic_operators.py |5 +- .../opperf/nd_operations/nn_conv_operators.py | 12 +- .../opperf/nd_operations/nn_optimizer_operators.py |4 +- .../nd_operations/random_sampling_operators.py |4 +- .../opperf/nd_operations/reduction_operators.py|4 +- .../nd_operations/sorting_searching_operators.py |4 +- benchmark/opperf/nd_operations/unary_operators.py |4 +- benchmark/opperf/opperf.py | 40 +- benchmark/opperf/utils/benchmark_utils.py | 27 +- benchmark/opperf/utils/common_utils.py | 38 +- benchmark/opperf/utils/ndarray_utils.py|4 - benchmark/opperf/utils/profiler_utils.py | 57 +- ci/jenkins/Jenkins_steps.groovy| 10 +- ci/jenkins/Jenkinsfile_unix_cpu|2 +- ci/publish/scala/build.sh |2 +- cmake/BuildTVM.cmake |2 +- contrib/clojure-package/.gitignore |1 + .../infer/objectdetector/images/marcel.jpg | Bin 116161 -> 0 bytes .../clojure-package/scripts/get_test_images.sh | 26 +- .../test/org/apache/clojure_mxnet/image_test.clj |6 +- .../clojure_mxnet/infer/imageclassifier_test.clj |5 +- .../test/test-images/Pug-Cookie.jpg| Bin 104323 -> 0 bytes .../clojure-package/test/test-images/kitten.jpg|
[GitHub] [incubator-mxnet] ZhennanQin opened a new pull request #15926: Add disable attr to subgraph property
ZhennanQin opened a new pull request #15926: Add disable attr to subgraph property URL: https://github.com/apache/incubator-mxnet/pull/15926 ## Description ## Can fix the crash after `export MXNET_DISABLE_MKLDNN_CONV_OPT=1` ## Checklist ## ### Essentials ### Please feel free to remove inapplicable items for your PR. - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes) - [ ] Changes are complete (i.e. I finished coding on this PR) - [ ] All changes have test coverage: - Unit tests are added for small changes to verify correctness (e.g. adding a new operator) - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore) - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL) - [ ] Code is well-documented: - For user-facing API changes, API doc string has been updated. - For new C++ functions in header files, their functionalities and arguments are documented. - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable - Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [ ] Feature1, tests, (and when applicable, API doc) - [ ] Feature2, tests, (and when applicable, API doc) ## Comments ## - If this change is a backward incompatible change, why must this change be made. - Interesting edge cases to note here This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout
zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521865680 Sure. @pengzhao-intel BTW, I have disabled MKLDNN subgraph backend to see whether it impacts on the efficiency of shuffle operator. The results showed the shuffle operator has the same time cost w/ and w/o MKLDNN subgraph backend. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] DickJC123 commented on issue #15922: Refactor for windows CI 'out of heap space' errors
DickJC123 commented on issue #15922: Refactor for windows CI 'out of heap space' errors URL: https://github.com/apache/incubator-mxnet/pull/15922#issuecomment-521865392 I reviewed the original laop_6 failure in one of my other PR CI runs and found it was not related to tolerances. Hence reverting the test tolerance change. The test_shuffle speed up commit does not alter tolerances or the functionality of the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] DickJC123 commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'.
DickJC123 commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'. URL: https://github.com/apache/incubator-mxnet/pull/15882#issuecomment-521863495 @larroy I looked again at the CI failure of laop_6: http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/windows-gpu/branches/PR-15882/runs/10/nodes/116/steps/186/log/?start=0. It appears to be unrelated to tolerances, so I reverted my change to that test. My change to test_shuffle was unrelated to tolerances- just sped up the checking code without altering any functionality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] DickJC123 commented on issue #13143: [MXNET-1206] Support NDArray indexing with None and Ellipsis
DickJC123 commented on issue #13143: [MXNET-1206] Support NDArray indexing with None and Ellipsis URL: https://github.com/apache/incubator-mxnet/pull/13143#issuecomment-521859970 @kohr-h A `git show` of this PR's commit includes: ``` diff --git a/python/mxnet/test_utils.py b/python/mxnet/test_utils.py index 2275f4d5d..89013f439 100644 --- a/python/mxnet/test_utils.py +++ b/python/mxnet/test_utils.py @@ -1074,7 +1074,11 @@ def check_symbolic_forward(sym, location, expected, rtol=1E-4, atol=None, executor = sym.bind(ctx=ctx, args=location, args_grad=args_grad_data, aux_states=aux_states) for g in executor.grad_arrays: -g[:] = 0 +print(g.shape) +if g.ndim == 0: +g[()] = 0 +else: +g[:] = 0 ``` Please confirm that the line `print(g.shape)` was a bit of debugging code inadvertently left in, and so should be removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] pengzhao-intel commented on issue #15880: [CI] unix cpu validation Timeout
pengzhao-intel commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521857381 Thanks @zixuanweeei Could we statistic and sort the runtime for all cases in CPU side (CPU, CPU+MKL, CPU+MKLDNN)? After that, we can see the runtime change by a new PR like @ChaiBapchya's large tensor PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch master updated (acc7865 -> c8f0b21)
This is an automated email from the ASF dual-hosted git repository. zachgk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from acc7865 added check for empty params file and unknown param (not arg/aux) (#15917) add c8f0b21 Remove Scala package test on build (#15915) No new revisions were added by this update. Summary of changes: ci/publish/scala/build.sh | 2 +- make/maven/maven_darwin_mkl.mk| 2 +- make/maven/maven_linux_cu90mkl.mk | 4 +++- make/maven/maven_linux_cu92mkl.mk | 4 +++- make/maven/maven_linux_mkl.mk | 2 +- make/pip/pip_linux_cu100.mk | 2 ++ make/pip/pip_linux_cu100mkl.mk| 2 ++ make/pip/pip_linux_cu101.mk | 2 ++ make/pip/pip_linux_cu101mkl.mk| 2 ++ make/pip/pip_linux_cu91.mk| 2 ++ make/pip/pip_linux_cu91mkl.mk | 2 ++ make/pip/pip_linux_cu92.mk| 2 ++ make/pip/pip_linux_cu92mkl.mk | 2 ++ tools/dependencies/openblas.sh| 2 +- tools/setup_gpu_build_tools.sh| 9 +++-- 15 files changed, 33 insertions(+), 8 deletions(-)
[GitHub] [incubator-mxnet] zachgk merged pull request #15915: Remove Scala package test on build
zachgk merged pull request #15915: Remove Scala package test on build URL: https://github.com/apache/incubator-mxnet/pull/15915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout
zixuanweeei commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521856502 @pengzhao-intel I've seen cpu runtimes more than 10 minutes by testing `test_random.py:test_shuffle` for three times. Seems there are lots of discussions on the shuffle operator, like PR #10048, PR #15882 and ISSUE #10277. I will take some surveys on them first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] xidulu commented on a change in pull request #15909: [numpy] random.rand
xidulu commented on a change in pull request #15909: [numpy] random.rand URL: https://github.com/apache/incubator-mxnet/pull/15909#discussion_r314559195 ## File path: python/mxnet/symbol/numpy/random.py ## @@ -49,6 +49,35 @@ def _random_helper(random, sampler, params, shape, dtype, ctx, out, kwargs): "but got %s." % type(params[0])) +def rand(*size, **kwargs): +r"""Random values in a given shape. + +Create an array of the given shape and populate it with random +samples from a uniform distribution over [0, 1). +Parameters +-- +d0, d1, ..., dn : int, optional +The dimensions of the returned array, should be all positive. +If no argument is given a single Python float is returned. +Returns +--- +out : ndarray +A ``(d0, d1, ..., dn)``-shaped array of floating-point samples from +the standard normal distribution, or a single such float if Review comment: Same as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] xidulu commented on a change in pull request #15909: [numpy] random.rand
xidulu commented on a change in pull request #15909: [numpy] random.rand URL: https://github.com/apache/incubator-mxnet/pull/15909#discussion_r314559079 ## File path: python/mxnet/numpy/random.py ## @@ -20,7 +20,36 @@ from __future__ import absolute_import from ..ndarray import numpy as _mx_nd_np -__all__ = ['uniform', 'normal'] +__all__ = ['uniform', 'normal', 'rand'] + + +def rand(*size, **kwargs): +r"""Random values in a given shape. + +Create an array of the given shape and populate it with random +samples from a uniform distribution over [0, 1). +Parameters +-- +d0, d1, ..., dn : int, optional +The dimensions of the returned array, should be all positive. +If no argument is given a single Python float is returned. +Returns +--- +out : ndarray +A ``(d0, d1, ..., dn)``-shaped array of floating-point samples from +the standard normal distribution, or a single such float if Review comment: Should remove "from the standard normal". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet] branch master updated (f586c07 -> acc7865)
This is an automated email from the ASF dual-hosted git repository. zachgk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git. from f586c07 [OpPerf] Profiler flag for Python, Cpp (#15881) add acc7865 added check for empty params file and unknown param (not arg/aux) (#15917) No new revisions were added by this update. Summary of changes: python/mxnet/model.py | 19 +-- 1 file changed, 13 insertions(+), 6 deletions(-)
[GitHub] [incubator-mxnet] zachgk merged pull request #15917: added check for empty params file and unknown param (not arg/aux)
zachgk merged pull request #15917: added check for empty params file and unknown param (not arg/aux) URL: https://github.com/apache/incubator-mxnet/pull/15917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] zachgk closed issue #15916: Error when loading models with no params
zachgk closed issue #15916: Error when loading models with no params URL: https://github.com/apache/incubator-mxnet/issues/15916 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #15925: [CI] illegal memory access
ChaiBapchya commented on issue #15925: [CI] illegal memory access URL: https://github.com/apache/incubator-mxnet/issues/15925#issuecomment-521852538 @mxnet-label-bot add [CI, Bug] This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya opened a new issue #15925: [CI] illegal memory access
ChaiBapchya opened a new issue #15925: [CI] illegal memory access URL: https://github.com/apache/incubator-mxnet/issues/15925 Multiple gpu tests fail due to illegal memory access PR - #15736 Pipeline - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-15736/9/pipeline ``` test_operator_gpu.test_np_flatten ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1294594193 to reproduce. ERROR test_operator_gpu.test_np_linspace ... [22:52:29] src/operator/tensor/./.././../common/../operator/mxnet_op.h:845: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:an illegal memory access was encountered /work/runtime_functions.sh: line 880: 6 Aborted (core dumped) nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] mxnet-label-bot commented on issue #15925: [CI] illegal memory access
mxnet-label-bot commented on issue #15925: [CI] illegal memory access URL: https://github.com/apache/incubator-mxnet/issues/15925#issuecomment-521852278 Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Bug, CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-site] branch asf-site updated: Bump the publish timestamp.
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 3db7aff Bump the publish timestamp. 3db7aff is described below commit 3db7aff02026e8362273d904ffd3da4833282e07 Author: mxnet-ci AuthorDate: Fri Aug 16 01:29:54 2019 + Bump the publish timestamp. --- date.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/date.txt b/date.txt new file mode 100644 index 000..54dee10 --- /dev/null +++ b/date.txt @@ -0,0 +1 @@ +Fri Aug 16 01:29:54 UTC 2019
[GitHub] [incubator-mxnet] zheng-da commented on issue #15898: missing numpy operators in MXNet.
zheng-da commented on issue #15898: missing numpy operators in MXNet. URL: https://github.com/apache/incubator-mxnet/issues/15898#issuecomment-521844731 `indices_or_sections` in `mx.np.split` should be int, tuple of int or a numpy array. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] zheng-da commented on issue #15898: missing numpy operators in MXNet.
zheng-da commented on issue #15898: missing numpy operators in MXNet. URL: https://github.com/apache/incubator-mxnet/issues/15898#issuecomment-521843801 numpy operators accept python list. For example, ``` np.cumsum([1, 2, 3, 4, 5]) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] pengzhao-intel commented on issue #15880: [CI] unix cpu validation Timeout
pengzhao-intel commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521826808 This is interesting and we need to figure out if the increased computation leads to the problem. @zixuanweeei could you help to take a look for the CI? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] marcoabreu commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'.
marcoabreu commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'. URL: https://github.com/apache/incubator-mxnet/pull/15882#issuecomment-521825065 Hi, there might be a misunderstanding. The issue here is not the cause for the timeouts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #9: Add GitHub bots
marcoabreu merged pull request #9: Add GitHub bots URL: https://github.com/apache/incubator-mxnet-ci/pull/9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch github-bots created (now b25d4e1)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch github-bots in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at b25d4e1 Add GitHub bots This branch includes the following new commits: new b25d4e1 Add GitHub bots The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #9: Add GitHub bots
marcoabreu opened a new pull request #9: Add GitHub bots URL: https://github.com/apache/incubator-mxnet-ci/pull/9 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch master updated: Add jenkins slave creation on windows
This is an automated email from the ASF dual-hosted git repository.
marcoabreu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git
The following commit(s) were added to refs/heads/master by this push:
new 953bb2e Add jenkins slave creation on windows
953bb2e is described below
commit 953bb2e162e77b3c631183556992882425f80503
Author: Marco de Abreu
AuthorDate: Fri Aug 16 00:25:05 2019 +0200
Add jenkins slave creation on windows
---
services/jenkins-slave-creation-windows/README.md | 18 +
.../conf-windows-cpu-c5/main.tf| 874 +
.../conf-windows-gpu-g3/main.tf| 871
.../infrastructure.tf | 258 ++
.../jenkins-slave-creation-windows/post-install.py | 280 +++
5 files changed, 2301 insertions(+)
diff --git a/services/jenkins-slave-creation-windows/README.md
b/services/jenkins-slave-creation-windows/README.md
new file mode 100644
index 000..6c6e39b
--- /dev/null
+++ b/services/jenkins-slave-creation-windows/README.md
@@ -0,0 +1,18 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Pipeline based on AWS SystemsManager to automatically created a Windows AMI to
be used as Jenkins slave every week.
\ No newline at end of file
diff --git
a/services/jenkins-slave-creation-windows/conf-windows-cpu-c5/main.tf
b/services/jenkins-slave-creation-windows/conf-windows-cpu-c5/main.tf
new file mode 100644
index 000..708956e
--- /dev/null
+++ b/services/jenkins-slave-creation-windows/conf-windows-cpu-c5/main.tf
@@ -0,0 +1,874 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "document_name" {}
+
+variable "source_ami_id" {}
+
+variable "iam_instance_profile_name" {}
+
+variable "automation_assume_role" {}
+
+variable "target_ami_name" {}
+
+variable "instance_type" {}
+
+variable "ebs_volume_size" {}
+
+variable "post_update_script_s3" {}
+
+variable "post_update_script_key" {}
+
+variable "post_update_script_path" {}
+
+variable "slave_autoconnect_python_s3" {}
+
+variable "slave_autoconnect_python_path" {}
+
+variable "slave_autoconnect_python_key" {}
+
+variable "slave_autoconnect_bat_s3" {}
+
+variable "slave_autoconnect_bat_path" {}
+
+variable "slave_autoconnect_bat_key" {}
+
+variable "cudnn_install_s3" {}
+
+variable "cudnn_install_key" {}
+
+variable "cudnn_install_path" {}
+
+resource "aws_ssm_document" "UpdateWindowsAMICPU" {
+ name = "${var.document_name}"
+ document_type = "Automation"
+
+ content =
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #8: Add unix slave creation
marcoabreu merged pull request #8: Add unix slave creation URL: https://github.com/apache/incubator-mxnet-ci/pull/8 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch master updated: Add unix slave creation (#8)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git The following commit(s) were added to refs/heads/master by this push: new 3990787 Add unix slave creation (#8) 3990787 is described below commit 3990787f6aa1fe2debb5f718bf91f1bf6d64805f Author: Marco de Abreu AuthorDate: Fri Aug 16 00:21:53 2019 +0200 Add unix slave creation (#8) --- tools/jenkins-slave-creation-unix/README.md| 18 ++ .../conf-ubuntu-cpu-c5/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-cpu-c5/install.sh | 128 .../conf-ubuntu-cpu-c5/shell-variables.sh | 22 ++ .../conf-ubuntu-gpu-g3/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-gpu-g3/install.sh | 144 + .../conf-ubuntu-gpu-g3/shell-variables.sh | 22 ++ .../conf-ubuntu-gpu-p3/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-gpu-p3/install.sh | 141 + .../conf-ubuntu-gpu-p3/shell-variables.sh | 22 ++ tools/jenkins-slave-creation-unix/create_slave.sh | 26 ++ .../jenkins-slave-creation-unix/infrastructure.tf | 326 tools/jenkins-slave-creation-unix/init.sh | 23 ++ .../scripts/create_slave.py| 106 +++ .../scripts/deploy/launch-autoconnect.sh | 22 ++ .../scripts/deploy/launch-ccache-mount-efs.sh | 40 +++ .../scripts/deploy/slave-autoconnect.py| 342 + 20 files changed, 1550 insertions(+) diff --git a/tools/jenkins-slave-creation-unix/README.md b/tools/jenkins-slave-creation-unix/README.md new file mode 100644 index 000..6b11f58 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/README.md @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + +This Terraform setup will spawn an instance that is ready to be saved into an AMI to create a Jenkins slave. \ No newline at end of file diff --git a/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars new file mode 100644 index 000..0aea681 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +key_name = "REDACTED" +key_path = "~/.ssh/REDACTED" +instance_type = "c5.18xlarge" + +additional_security_group_ids = [ + "sg-5d83d421", # VPC default + "sg-REDACTED" # REDACTED +] + +s3_config_bucket = "mxnet-ci-slave-dev" +s3_config_filename = "ubuntu-cpu-c5-config.tar.bz2" +slave_install_script = "conf-ubuntu-cpu-c5/install.sh" +shell_variables_file = "conf-ubuntu-cpu-c5/shell-variables.sh" +ami = "ami-bd8f33c5" # ftp://64.50.236.216/pub/ubuntu-cloud-images/query/xenial/server/released.txt +instance_name = "Slave-base_Ubuntu-CPU-C5" +aws_region = "us-west-2" +secret_manager_docker_hub_arn = "arn:aws:secretsmanager:us-west-2:REDACTED:secret:REDACTED" diff --git a/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars new file mode 100644 index 000..3e07798 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR
[incubator-mxnet-ci] branch unix-slave-creation created (now 628483f)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch unix-slave-creation in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at 628483f Add unix slave creation This branch includes the following new commits: new 628483f Add unix slave creation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[incubator-mxnet-ci] 01/01: Add unix slave creation
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch unix-slave-creation in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git commit 628483f16f24f0d07d2172b4f4849ef524c8e4a9 Author: Marco de Abreu AuthorDate: Fri Aug 16 00:17:18 2019 +0200 Add unix slave creation --- tools/jenkins-slave-creation-unix/README.md| 18 ++ .../conf-ubuntu-cpu-c5/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-cpu-c5/install.sh | 128 .../conf-ubuntu-cpu-c5/shell-variables.sh | 22 ++ .../conf-ubuntu-gpu-g3/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-gpu-g3/install.sh | 144 + .../conf-ubuntu-gpu-g3/shell-variables.sh | 22 ++ .../conf-ubuntu-gpu-p3/infrastructure.tfvars | 34 ++ .../infrastructure_backend.tfvars | 22 ++ .../conf-ubuntu-gpu-p3/install.sh | 141 + .../conf-ubuntu-gpu-p3/shell-variables.sh | 22 ++ tools/jenkins-slave-creation-unix/create_slave.sh | 26 ++ .../jenkins-slave-creation-unix/infrastructure.tf | 326 tools/jenkins-slave-creation-unix/init.sh | 23 ++ .../scripts/create_slave.py| 106 +++ .../scripts/deploy/launch-autoconnect.sh | 22 ++ .../scripts/deploy/launch-ccache-mount-efs.sh | 40 +++ .../scripts/deploy/slave-autoconnect.py| 342 + 20 files changed, 1550 insertions(+) diff --git a/tools/jenkins-slave-creation-unix/README.md b/tools/jenkins-slave-creation-unix/README.md new file mode 100644 index 000..6b11f58 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/README.md @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + +This Terraform setup will spawn an instance that is ready to be saved into an AMI to create a Jenkins slave. \ No newline at end of file diff --git a/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars new file mode 100644 index 000..0aea681 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure.tfvars @@ -0,0 +1,34 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +key_name = "REDACTED" +key_path = "~/.ssh/REDACTED" +instance_type = "c5.18xlarge" + +additional_security_group_ids = [ + "sg-5d83d421", # VPC default + "sg-REDACTED" # REDACTED +] + +s3_config_bucket = "mxnet-ci-slave-dev" +s3_config_filename = "ubuntu-cpu-c5-config.tar.bz2" +slave_install_script = "conf-ubuntu-cpu-c5/install.sh" +shell_variables_file = "conf-ubuntu-cpu-c5/shell-variables.sh" +ami = "ami-bd8f33c5" # ftp://64.50.236.216/pub/ubuntu-cloud-images/query/xenial/server/released.txt +instance_name = "Slave-base_Ubuntu-CPU-C5" +aws_region = "us-west-2" +secret_manager_docker_hub_arn = "arn:aws:secretsmanager:us-west-2:REDACTED:secret:REDACTED" diff --git a/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars new file mode 100644 index 000..3e07798 --- /dev/null +++ b/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/infrastructure_backend.tfvars @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +#
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #8: Add unix slave creation
marcoabreu opened a new pull request #8: Add unix slave creation URL: https://github.com/apache/incubator-mxnet-ci/pull/8 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] haojin2 opened a new pull request #15924: Numpy-compatible cumsum upstream
haojin2 opened a new pull request #15924: Numpy-compatible cumsum upstream URL: https://github.com/apache/incubator-mxnet/pull/15924 ## Description ## As title. ## Checklist ## ### Essentials ### Please feel free to remove inapplicable items for your PR. - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes) - [ ] Changes are complete (i.e. I finished coding on this PR) - [ ] All changes have test coverage: - Unit tests are added for small changes to verify correctness (e.g. adding a new operator) - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore) - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL) - [ ] Code is well-documented: - For user-facing API changes, API doc string has been updated. - For new C++ functions in header files, their functionalities and arguments are documented. - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable - Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [x] Forward and Backward - [x] Unit tests - [x] also added for `nd` and `sym` ## Comments ## @ChaiBapchya This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch master updated (12de57a -> 3d74ae8)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. discard 12de57a Add jenkins master (#7) omit 8ad0056 Add Jenkins Autoscaling (#6) new e23f5ff Add Jenkins Autoscaling (#6) new 3d74ae8 Add jenkins master (#7) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (12de57a) \ N -- N -- N refs/heads/master (3d74ae8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: services/jenkins-autoscaling/lambda_mxnet_ci/autoscaling/handler.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[incubator-mxnet-ci] 02/02: Add jenkins master (#7)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git commit 3d74ae8b3f13dff5f3748d662cddfd746422d9ec Author: Marco de Abreu AuthorDate: Fri Aug 16 00:03:34 2019 +0200 Add jenkins master (#7) --- services/jenkins-master/.gitignore | 6 + services/jenkins-master/README.md | 35 ++ services/jenkins-master/deploy.sh | 23 ++ services/jenkins-master/download_config.sh | 23 ++ services/jenkins-master/infrastructure.tf | 351 + services/jenkins-master/prod/infrastructure.tfvars | 33 ++ .../jenkins-master/prod/jenkins/REDACTED-FULLY | 0 .../scripts/deploy_infrastructure.py | 186 +++ services/jenkins-master/scripts/docker_setup.sh| 163 ++ .../scripts/jenkins_config_templating.py | 275 services/jenkins-master/scripts/jenkins_setup.sh | 120 +++ .../jenkins-master/scripts/jenkins_sync_config.py | 134 services/jenkins-master/scripts/sync_ci_to_host.sh | 48 +++ services/jenkins-master/scripts/sync_host_to_ci.sh | 54 services/jenkins-master/test/infrastructure.tfvars | 39 +++ .../test/infrastructure_backend.tfvars | 22 ++ .../jenkins-master/test/jenkins/REDACYED-FULLY | 0 .../jenkins-master/test/jenkins_config.symlinkfile | 55 .../jenkins-master/test/jenkins_config.varfile | 8 + .../jenkins-master/test/secrets/REDACTED-FULLY | 0 services/jenkins-master/test/variables.sh | 21 ++ 21 files changed, 1596 insertions(+) diff --git a/services/jenkins-master/.gitignore b/services/jenkins-master/.gitignore new file mode 100644 index 000..fda640c --- /dev/null +++ b/services/jenkins-master/.gitignore @@ -0,0 +1,6 @@ +.terraform/ +jenkins.tar.bz2 +.terraform.tfstate.lock.info +__pycache__/ +jenkins_plugins.tar.bz2 +temp/ diff --git a/services/jenkins-master/README.md b/services/jenkins-master/README.md new file mode 100644 index 000..631da0b --- /dev/null +++ b/services/jenkins-master/README.md @@ -0,0 +1,35 @@ + + + + + + + + + + + + + + + + + +This is the Terraform setup we used to create the Jenkins master. This script is **VERY** outdated! + +# Create infrastructure + +Warning: this will destroy the current DNS entries. + +``` +./init.sh +terraform init +terraform apply +``` + + +With difference instance type (overriding variables) + +``` +terraform apply -var instance_type=c1.xlarge +``` diff --git a/services/jenkins-master/deploy.sh b/services/jenkins-master/deploy.sh new file mode 100755 index 000..8ef0e04 --- /dev/null +++ b/services/jenkins-master/deploy.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3 scripts/deploy_infrastructure.py --configdir "$config_dir" --terraformdir "." diff --git a/services/jenkins-master/download_config.sh b/services/jenkins-master/download_config.sh new file mode 100755 index 000..0d29ede --- /dev/null +++ b/services/jenkins-master/download_config.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3 scripts/jenkins_sync_config.py --jenkinsdir "$config_dir/jenkins" --varfile "$config_dir/jenkins_config.varfile" --symlinkfile
[incubator-mxnet-ci] branch master updated: Add jenkins master (#7)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git The following commit(s) were added to refs/heads/master by this push: new 12de57a Add jenkins master (#7) 12de57a is described below commit 12de57a192f462e07446986b24bd2bd03edb06f6 Author: Marco de Abreu AuthorDate: Fri Aug 16 00:03:34 2019 +0200 Add jenkins master (#7) --- services/jenkins-master/.gitignore | 6 + services/jenkins-master/README.md | 35 ++ services/jenkins-master/deploy.sh | 23 ++ services/jenkins-master/download_config.sh | 23 ++ services/jenkins-master/infrastructure.tf | 351 + services/jenkins-master/prod/infrastructure.tfvars | 33 ++ .../jenkins-master/prod/jenkins/REDACTED-FULLY | 0 .../scripts/deploy_infrastructure.py | 186 +++ services/jenkins-master/scripts/docker_setup.sh| 163 ++ .../scripts/jenkins_config_templating.py | 275 services/jenkins-master/scripts/jenkins_setup.sh | 120 +++ .../jenkins-master/scripts/jenkins_sync_config.py | 134 services/jenkins-master/scripts/sync_ci_to_host.sh | 48 +++ services/jenkins-master/scripts/sync_host_to_ci.sh | 54 services/jenkins-master/test/infrastructure.tfvars | 39 +++ .../test/infrastructure_backend.tfvars | 22 ++ .../jenkins-master/test/jenkins/REDACYED-FULLY | 0 .../jenkins-master/test/jenkins_config.symlinkfile | 55 .../jenkins-master/test/jenkins_config.varfile | 8 + .../jenkins-master/test/secrets/REDACTED-FULLY | 0 services/jenkins-master/test/variables.sh | 21 ++ 21 files changed, 1596 insertions(+) diff --git a/services/jenkins-master/.gitignore b/services/jenkins-master/.gitignore new file mode 100644 index 000..fda640c --- /dev/null +++ b/services/jenkins-master/.gitignore @@ -0,0 +1,6 @@ +.terraform/ +jenkins.tar.bz2 +.terraform.tfstate.lock.info +__pycache__/ +jenkins_plugins.tar.bz2 +temp/ diff --git a/services/jenkins-master/README.md b/services/jenkins-master/README.md new file mode 100644 index 000..631da0b --- /dev/null +++ b/services/jenkins-master/README.md @@ -0,0 +1,35 @@ + + + + + + + + + + + + + + + + + +This is the Terraform setup we used to create the Jenkins master. This script is **VERY** outdated! + +# Create infrastructure + +Warning: this will destroy the current DNS entries. + +``` +./init.sh +terraform init +terraform apply +``` + + +With difference instance type (overriding variables) + +``` +terraform apply -var instance_type=c1.xlarge +``` diff --git a/services/jenkins-master/deploy.sh b/services/jenkins-master/deploy.sh new file mode 100755 index 000..8ef0e04 --- /dev/null +++ b/services/jenkins-master/deploy.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3 scripts/deploy_infrastructure.py --configdir "$config_dir" --terraformdir "." diff --git a/services/jenkins-master/download_config.sh b/services/jenkins-master/download_config.sh new file mode 100755 index 000..0d29ede --- /dev/null +++ b/services/jenkins-master/download_config.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #7: Add jenkins master
marcoabreu merged pull request #7: Add jenkins master URL: https://github.com/apache/incubator-mxnet-ci/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch jenkins-master created (now 0dddb7f)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch jenkins-master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at 0dddb7f Add jenkins master This branch includes the following new commits: new 0dddb7f Add jenkins master The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #7: Add jenkins master
marcoabreu opened a new pull request #7: Add jenkins master URL: https://github.com/apache/incubator-mxnet-ci/pull/7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] 01/01: Add jenkins master
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch jenkins-master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git commit 0dddb7f9eedad5a85e29626ad1cb0f99fa78b7d7 Author: Marco de Abreu AuthorDate: Thu Aug 15 23:58:22 2019 +0200 Add jenkins master --- services/jenkins-master/.gitignore | 6 + services/jenkins-master/README.md | 35 ++ services/jenkins-master/deploy.sh | 23 ++ services/jenkins-master/download_config.sh | 23 ++ services/jenkins-master/infrastructure.tf | 351 + services/jenkins-master/prod/infrastructure.tfvars | 33 ++ .../jenkins-master/prod/jenkins/REDACTED-FULLY | 0 .../scripts/deploy_infrastructure.py | 186 +++ services/jenkins-master/scripts/docker_setup.sh| 163 ++ .../scripts/jenkins_config_templating.py | 275 services/jenkins-master/scripts/jenkins_setup.sh | 120 +++ .../jenkins-master/scripts/jenkins_sync_config.py | 134 services/jenkins-master/scripts/sync_ci_to_host.sh | 48 +++ services/jenkins-master/scripts/sync_host_to_ci.sh | 54 services/jenkins-master/test/infrastructure.tfvars | 39 +++ .../test/infrastructure_backend.tfvars | 22 ++ .../jenkins-master/test/jenkins/REDACYED-FULLY | 0 .../jenkins-master/test/jenkins_config.symlinkfile | 55 .../jenkins-master/test/jenkins_config.varfile | 8 + .../jenkins-master/test/secrets/REDACTED-FULLY | 0 services/jenkins-master/test/variables.sh | 21 ++ 21 files changed, 1596 insertions(+) diff --git a/services/jenkins-master/.gitignore b/services/jenkins-master/.gitignore new file mode 100644 index 000..fda640c --- /dev/null +++ b/services/jenkins-master/.gitignore @@ -0,0 +1,6 @@ +.terraform/ +jenkins.tar.bz2 +.terraform.tfstate.lock.info +__pycache__/ +jenkins_plugins.tar.bz2 +temp/ diff --git a/services/jenkins-master/README.md b/services/jenkins-master/README.md new file mode 100644 index 000..631da0b --- /dev/null +++ b/services/jenkins-master/README.md @@ -0,0 +1,35 @@ + + + + + + + + + + + + + + + + + +This is the Terraform setup we used to create the Jenkins master. This script is **VERY** outdated! + +# Create infrastructure + +Warning: this will destroy the current DNS entries. + +``` +./init.sh +terraform init +terraform apply +``` + + +With difference instance type (overriding variables) + +``` +terraform apply -var instance_type=c1.xlarge +``` diff --git a/services/jenkins-master/deploy.sh b/services/jenkins-master/deploy.sh new file mode 100755 index 000..8ef0e04 --- /dev/null +++ b/services/jenkins-master/deploy.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3 scripts/deploy_infrastructure.py --configdir "$config_dir" --terraformdir "." diff --git a/services/jenkins-master/download_config.sh b/services/jenkins-master/download_config.sh new file mode 100755 index 000..0d29ede --- /dev/null +++ b/services/jenkins-master/download_config.sh @@ -0,0 +1,23 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +echo "Config dir:" +read config_dir + +python3 scripts/jenkins_sync_config.py --jenkinsdir "$config_dir/jenkins" --varfile "$config_dir/jenkins_config.varfile" --symlinkfile
[GitHub] [incubator-mxnet] eric-haibin-lin opened a new issue #15923: kvstore.save_optimizer_states does not support distributed kvstore
eric-haibin-lin opened a new issue #15923: kvstore.save_optimizer_states does not support distributed kvstore URL: https://github.com/apache/incubator-mxnet/issues/15923 States cannot be saved during distributed training: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/kvstore.py#L538-L550 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] mxnet-label-bot commented on issue #15923: kvstore.save_optimizer_states does not support distributed kvstore
mxnet-label-bot commented on issue #15923: kvstore.save_optimizer_states does not support distributed kvstore URL: https://github.com/apache/incubator-mxnet/issues/15923#issuecomment-521811897 Hey, this is the MXNet Label Bot. Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it. Here are my recommended labels: Feature This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] vandanavk commented on issue #15914: Correct ONNX documentation
vandanavk commented on issue #15914: Correct ONNX documentation URL: https://github.com/apache/incubator-mxnet/pull/15914#issuecomment-521811587 @IvyBazan for review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #6: Add Jenkins Autoscaling
marcoabreu merged pull request #6: Add Jenkins Autoscaling URL: https://github.com/apache/incubator-mxnet-ci/pull/6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch autoscaling created (now a523aa5)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch autoscaling in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at a523aa5 Add Jenkins Autoscaling This branch includes the following new commits: new a523aa5 Add Jenkins Autoscaling The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #6: Add Jenkins Autoscaling
marcoabreu opened a new pull request #6: Add Jenkins Autoscaling URL: https://github.com/apache/incubator-mxnet-ci/pull/6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] haojin2 merged pull request #15903: [Numpy] Random.randn() implemented.
haojin2 merged pull request #15903: [Numpy] Random.randn() implemented. URL: https://github.com/apache/incubator-mxnet/pull/15903 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch master updated: Add rotate secrets manager credentials (#5)
This is an automated email from the ASF dual-hosted git repository.
marcoabreu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git
The following commit(s) were added to refs/heads/master by this push:
new e5f051a Add rotate secrets manager credentials (#5)
e5f051a is described below
commit e5f051a2a09e4ec20627b396117e202cced0df7b
Author: Marco de Abreu
AuthorDate: Thu Aug 15 23:29:26 2019 +0200
Add rotate secrets manager credentials (#5)
---
.../rotate-secrets-manager-credentials/.gitignore | 47 +++
tools/rotate-secrets-manager-credentials/README.md | 39 +++
.../deploy_lambda.sh | 36 +++
.../docker_hub_change_password.py | 342 +
.../environment.yml| 8 +
.../requirements.txt | 1 +
.../serverless.yml | 76 +
7 files changed, 549 insertions(+)
diff --git a/tools/rotate-secrets-manager-credentials/.gitignore
b/tools/rotate-secrets-manager-credentials/.gitignore
new file mode 100755
index 000..1bb0ffd
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/.gitignore
@@ -0,0 +1,47 @@
+*~
+package-lock.json
+package.json
+
+# Logs
+logs
+*.log
+npm-debug.log
+
+# Runtime data
+pids
+*.pid
+*.seed
+dist
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# Grunt intermediate storage
(http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directory
+#
https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
+node_modules
+
+#IDE Stuff
+**/.idea
+
+#OS STUFF
+.DS_Store
+.tmp
+
+#SERVERLESS STUFF
+admin.env
+.env
+_meta
+.serverless
+
diff --git a/tools/rotate-secrets-manager-credentials/README.md
b/tools/rotate-secrets-manager-credentials/README.md
new file mode 100644
index 000..52609d7
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/README.md
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# Lambda function for automatic rotatation of DockerHub credentials in AWS
SecretsManager
+This repository sets up a Lambda function that allows to autoamtically change
a DockerHub password.
+
+## Deployment
+These are the deployment instructions.
+
+### Installation
+You need to have NodeJS (npm) and serverless installed. Additional npm
packages are required and can be installed as follows:
+```npm install serverless-python-requirements```
+```npm install serverless-s3-remover```
+
+### Provisioning
+Run ```deploy_lambda.sh``` and enter the deployment stage
+
+## Usage
+Log into SecretsManager, open the secret of your choice, go to the category
"Rotation configuration", click on "Edit rotation", enable the automatic
rotation, enter an interval of your choice and select the previously
provisioned Lambda function. Then press on "Save"; note that this will trigger
an immediate rotation of the credentials.
+
+If you would like to trigger a manual immediate rotation, click on "Rotate
secret immediately" in the secret detail windows.
+
+## Debugging
+If you'd like to debug this script, go to CloudWatch logs and look for the "
+/aws/lambda/SecretsManager_docker_hub_change_password_function" log group.
\ No newline at end of file
diff --git a/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
b/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
new file mode 100755
index 000..269a646
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+echo "Deployment stage (test, prod)"
+read config_dir
+
+if [ "$config_dir" == "test" ]; then
+echo "Deploying to test"
+export AWS_PROFILE=mxnet-ci-dev
+sls deploy -s test
+elif [ "$config_dir" == "prod" ]; then
+echo "Deploying to prod"
+export AWS_PROFILE=mxnet-ci
+sls deploy -s prod
+else
+echo "Unrecognized stage: ${config_dir}"
+fi
+
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #5: Add rotate secrets manager credentials
marcoabreu opened a new pull request #5: Add rotate secrets manager credentials URL: https://github.com/apache/incubator-mxnet-ci/pull/5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #5: Add rotate secrets manager credentials
marcoabreu merged pull request #5: Add rotate secrets manager credentials URL: https://github.com/apache/incubator-mxnet-ci/pull/5 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch rotate-secrets-manager-credentials created (now c8f2ed4)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch rotate-secrets-manager-credentials in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at c8f2ed4 Add rotate secrets manager credentials This branch includes the following new commits: new c8f2ed4 Add rotate secrets manager credentials The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[incubator-mxnet-ci] 01/01: Add rotate secrets manager credentials
This is an automated email from the ASF dual-hosted git repository.
marcoabreu pushed a commit to branch rotate-secrets-manager-credentials
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git
commit c8f2ed48234d3b21f233d2e95d3a5edf3f440627
Author: Marco de Abreu
AuthorDate: Thu Aug 15 23:25:17 2019 +0200
Add rotate secrets manager credentials
---
.../rotate-secrets-manager-credentials/.gitignore | 47 +++
tools/rotate-secrets-manager-credentials/README.md | 39 +++
.../deploy_lambda.sh | 36 +++
.../docker_hub_change_password.py | 342 +
.../environment.yml| 8 +
.../requirements.txt | 1 +
.../serverless.yml | 76 +
7 files changed, 549 insertions(+)
diff --git a/tools/rotate-secrets-manager-credentials/.gitignore
b/tools/rotate-secrets-manager-credentials/.gitignore
new file mode 100755
index 000..1bb0ffd
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/.gitignore
@@ -0,0 +1,47 @@
+*~
+package-lock.json
+package.json
+
+# Logs
+logs
+*.log
+npm-debug.log
+
+# Runtime data
+pids
+*.pid
+*.seed
+dist
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# Grunt intermediate storage
(http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directory
+#
https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
+node_modules
+
+#IDE Stuff
+**/.idea
+
+#OS STUFF
+.DS_Store
+.tmp
+
+#SERVERLESS STUFF
+admin.env
+.env
+_meta
+.serverless
+
diff --git a/tools/rotate-secrets-manager-credentials/README.md
b/tools/rotate-secrets-manager-credentials/README.md
new file mode 100644
index 000..52609d7
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/README.md
@@ -0,0 +1,39 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# Lambda function for automatic rotatation of DockerHub credentials in AWS
SecretsManager
+This repository sets up a Lambda function that allows to autoamtically change
a DockerHub password.
+
+## Deployment
+These are the deployment instructions.
+
+### Installation
+You need to have NodeJS (npm) and serverless installed. Additional npm
packages are required and can be installed as follows:
+```npm install serverless-python-requirements```
+```npm install serverless-s3-remover```
+
+### Provisioning
+Run ```deploy_lambda.sh``` and enter the deployment stage
+
+## Usage
+Log into SecretsManager, open the secret of your choice, go to the category
"Rotation configuration", click on "Edit rotation", enable the automatic
rotation, enter an interval of your choice and select the previously
provisioned Lambda function. Then press on "Save"; note that this will trigger
an immediate rotation of the credentials.
+
+If you would like to trigger a manual immediate rotation, click on "Rotate
secret immediately" in the secret detail windows.
+
+## Debugging
+If you'd like to debug this script, go to CloudWatch logs and look for the "
+/aws/lambda/SecretsManager_docker_hub_change_password_function" log group.
\ No newline at end of file
diff --git a/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
b/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
new file mode 100755
index 000..269a646
--- /dev/null
+++ b/tools/rotate-secrets-manager-credentials/deploy_lambda.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+echo "Deployment stage (test, prod)"
+read config_dir
+
+if [ "$config_dir" == "test" ]; then
+echo "Deploying to test"
+export AWS_PROFILE=mxnet-ci-dev
+sls deploy -s test
+elif [ "$config_dir" == "prod" ]; then
+echo "Deploying to prod"
+export AWS_PROFILE=mxnet-ci
+sls deploy -s prod
+else
+echo "Unrecognized stage: ${config_dir}"
+fi
+
diff --git
a/tools/rotate-secrets-manager-credentials/docker_hub_change_password.py
[GitHub] [incubator-mxnet] DickJC123 opened a new pull request #15922: Refactor for windows CI 'out of heap space' errors
DickJC123 opened a new pull request #15922: Refactor for windows CI 'out of
heap space' errors
URL: https://github.com/apache/incubator-mxnet/pull/15922
## Description ##
This PR can be thought of as an alternate proposal to PR
https://github.com/apache/incubator-mxnet/pull/15882 as far as how to handle
the 'out of heap space' errors of the windows CI. This PR follows the
suggestion of @marcoabreu to break up the problematic large-compiles into
smaller pieces, rather than moving the Windows compiler toolchain to 64-bit.
So far the PR only breaks apart
./src/operator/tensor/broadcast_reduce_op_value.{cu,cc}. The PR also
cherry-picks a test_shuffle speed-up and a laop_6 flakiness fix to help get the
PR through CI.
The pitch: smaller compiler tasks are more easily parallelized over the CPU
cores in a 'make -j N' scenario, though it's not clear if total compile time is
reduced. It may decrease the time needed for a failing compile to complete- a
help for those that like to isolate a current compile problem with a follow-up
simple 'make' command.
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to
the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues)
created (except PRs with tiny changes)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ X] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding
a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing
distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a
new build option with NCCL)
- [X ] Code is well-documented:
- For user-facing API changes, API doc string has been updated.
- For new C++ functions in header files, their functionalities and arguments
are documented.
- For new examples, README.md is added to explain the what the example does,
the source of the dataset, expected performance on test set and reference to
the original paper if applicable
- Check the API doc at
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
- [ X] To the my best knowledge, examples are either not affected by this
change, or have been fixed to be compatible with this change
### Changes ###
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
## Comments ##
- If this change is a backward incompatible change, why must this change be
made.
- Interesting edge cases to note here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #15880: [CI] unix cpu validation Timeout
ChaiBapchya commented on issue #15880: [CI] unix cpu validation Timeout URL: https://github.com/apache/incubator-mxnet/issues/15880#issuecomment-521803473 Another PR #15541 Python 3 CPU (runs for 4hours) before terminating! http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-15541/8/pipeline/279 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland commented on issue #15596: Unable to convert reference model to ONNX
KellenSunderland commented on issue #15596: Unable to convert reference model to ONNX URL: https://github.com/apache/incubator-mxnet/issues/15596#issuecomment-521803510 Thanks @vandanavk! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] KellenSunderland closed issue #15596: Unable to convert reference model to ONNX
KellenSunderland closed issue #15596: Unable to convert reference model to ONNX URL: https://github.com/apache/incubator-mxnet/issues/15596 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet-ci] marcoabreu merged pull request #4: Add jenkins run statistics
marcoabreu merged pull request #4: Add jenkins run statistics URL: https://github.com/apache/incubator-mxnet-ci/pull/4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] vandanavk commented on issue #15596: Unable to convert reference model to ONNX
vandanavk commented on issue #15596: Unable to convert reference model to ONNX URL: https://github.com/apache/incubator-mxnet/issues/15596#issuecomment-521803254 try adding batch_shape within [] in the export_model() call. `onnx_mxnet.export_model(sym=sym, input_shape=[batch_shape], verbose=True, params=all_params) ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet-ci] marcoabreu opened a new pull request #4: Add jenkins run statistics
marcoabreu opened a new pull request #4: Add jenkins run statistics URL: https://github.com/apache/incubator-mxnet-ci/pull/4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-mxnet-ci] branch master updated: Add jenkins run statistics (#4)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git The following commit(s) were added to refs/heads/master by this push: new e1e4dc4 Add jenkins run statistics (#4) e1e4dc4 is described below commit e1e4dc4d9a042ba1466487151eab132988bc04ae Author: Marco de Abreu AuthorDate: Thu Aug 15 23:25:36 2019 +0200 Add jenkins run statistics (#4) --- services/jenkins-run-statistics/.gitignore | 46 services/jenkins-run-statistics/README.md| 37 +++ services/jenkins-run-statistics/__init__.py | 0 services/jenkins-run-statistics/aws_utils.py | 69 ++ services/jenkins-run-statistics/deploy_lambda.sh | 35 +++ services/jenkins-run-statistics/environment.yml | 12 + services/jenkins-run-statistics/jenkins_utils.py | 259 +++ services/jenkins-run-statistics/requirements.txt | 3 + services/jenkins-run-statistics/serverless.yml | 77 ++ services/jenkins-run-statistics/statistics.py| 300 +++ 10 files changed, 838 insertions(+) diff --git a/services/jenkins-run-statistics/.gitignore b/services/jenkins-run-statistics/.gitignore new file mode 100644 index 000..c021710 --- /dev/null +++ b/services/jenkins-run-statistics/.gitignore @@ -0,0 +1,46 @@ +*~ +__pycache__/ + +# Logs +logs +*.log +npm-debug.log + +# Runtime data +pids +*.pid +*.seed +dist + +# Directory for instrumented libs generated by jscoverage/JSCover +lib-cov + +# Coverage directory used by tools like istanbul +coverage + +# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) +.grunt + +# node-waf configuration +.lock-wscript + +# Compiled binary addons (http://nodejs.org/api/addons.html) +build/Release + +# Dependency directory +# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git +node_modules +package-lock.json + +#IDE Stuff +**/.idea + +#OS STUFF +.DS_Store +.tmp + +#SERVERLESS STUFF +admin.env +.env +_meta +.serverless diff --git a/services/jenkins-run-statistics/README.md b/services/jenkins-run-statistics/README.md new file mode 100644 index 000..c6d6e38 --- /dev/null +++ b/services/jenkins-run-statistics/README.md @@ -0,0 +1,37 @@ + + + + + + + + + + + + + + + + + +# Jenkins run statistics +This script automatically generates CloudWatch metrics regarding the duration of Jenkins runs. + +## Metrics +The metrics can be found in CloudWatch metrics. Check the environment.yml for the metric namespace. + +## Logs +The logs are available in CloudWatch logs. Check the serverless.yml for the log namespace. + +## Limitations +This tool processes all runs that are in the Jenkins database, but CloudWatch Metrics only allows to go back as far as 14 days. Thus, any runs that are older will be skipped. Please also note that for metrics, that are older than 24 hours, it may take them up to 48 hours until they are visible in the web interface. Consult https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Metric and https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API [...] + +If this lambda function times out due to too much data, it will automatically recover from that state and continue the work at the same point. This is achieved due to the DynamoDB backend in combination with Jenkins reporting the last build time of each job. + +## Set up +- Install the Serverless framework + +## Execution +Run deploy_lambda.sh + diff --git a/services/jenkins-run-statistics/__init__.py b/services/jenkins-run-statistics/__init__.py new file mode 100644 index 000..e69de29 diff --git a/services/jenkins-run-statistics/aws_utils.py b/services/jenkins-run-statistics/aws_utils.py new file mode 100644 index 000..f0a1628 --- /dev/null +++ b/services/jenkins-run-statistics/aws_utils.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import time +from datetime import datetime, timedelta, timezone +import logging +from collections import namedtuple + +import boto3 +from botocore.exceptions import
[incubator-mxnet-ci] 01/01: Add jenkins run statistics
This is an automated email from the ASF dual-hosted git repository.
marcoabreu pushed a commit to branch jenkins-run-statistics
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git
commit b7c72dea081ec2d687b96ef2e0caf6ec14d8da29
Author: Marco de Abreu
AuthorDate: Thu Aug 15 23:21:32 2019 +0200
Add jenkins run statistics
---
services/jenkins-run-statistics/.gitignore | 46
services/jenkins-run-statistics/README.md| 37 +++
services/jenkins-run-statistics/__init__.py | 0
services/jenkins-run-statistics/aws_utils.py | 69 ++
services/jenkins-run-statistics/deploy_lambda.sh | 35 +++
services/jenkins-run-statistics/environment.yml | 12 +
services/jenkins-run-statistics/jenkins_utils.py | 259 +++
services/jenkins-run-statistics/requirements.txt | 3 +
services/jenkins-run-statistics/serverless.yml | 77 ++
services/jenkins-run-statistics/statistics.py| 300 +++
10 files changed, 838 insertions(+)
diff --git a/services/jenkins-run-statistics/.gitignore
b/services/jenkins-run-statistics/.gitignore
new file mode 100644
index 000..c021710
--- /dev/null
+++ b/services/jenkins-run-statistics/.gitignore
@@ -0,0 +1,46 @@
+*~
+__pycache__/
+
+# Logs
+logs
+*.log
+npm-debug.log
+
+# Runtime data
+pids
+*.pid
+*.seed
+dist
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# Grunt intermediate storage
(http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directory
+#
https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
+node_modules
+package-lock.json
+
+#IDE Stuff
+**/.idea
+
+#OS STUFF
+.DS_Store
+.tmp
+
+#SERVERLESS STUFF
+admin.env
+.env
+_meta
+.serverless
diff --git a/services/jenkins-run-statistics/README.md
b/services/jenkins-run-statistics/README.md
new file mode 100644
index 000..c6d6e38
--- /dev/null
+++ b/services/jenkins-run-statistics/README.md
@@ -0,0 +1,37 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+# Jenkins run statistics
+This script automatically generates CloudWatch metrics regarding the duration
of Jenkins runs.
+
+## Metrics
+The metrics can be found in CloudWatch metrics. Check the environment.yml for
the metric namespace.
+
+## Logs
+The logs are available in CloudWatch logs. Check the serverless.yml for the
log namespace.
+
+## Limitations
+This tool processes all runs that are in the Jenkins database, but CloudWatch
Metrics only allows to go back as far as 14 days. Thus, any runs that are older
will be skipped. Please also note that for metrics, that are older than 24
hours, it may take them up to 48 hours until they are visible in the web
interface. Consult
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Metric
and https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API [...]
+
+If this lambda function times out due to too much data, it will automatically
recover from that state and continue the work at the same point. This is
achieved due to the DynamoDB backend in combination with Jenkins reporting the
last build time of each job.
+
+## Set up
+- Install the Serverless framework
+
+## Execution
+Run deploy_lambda.sh
+
diff --git a/services/jenkins-run-statistics/__init__.py
b/services/jenkins-run-statistics/__init__.py
new file mode 100644
index 000..e69de29
diff --git a/services/jenkins-run-statistics/aws_utils.py
b/services/jenkins-run-statistics/aws_utils.py
new file mode 100644
index 000..f0a1628
--- /dev/null
+++ b/services/jenkins-run-statistics/aws_utils.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import time
+from datetime import datetime, timedelta, timezone
+import logging
+from collections import namedtuple
+
+import boto3
+from botocore.exceptions import ClientError
+
+
+AwsServiceObjectsTuple = namedtuple('AwsServiceObjectsTuple', ['dynamo_db',
'cloudwatch'])
+
[incubator-mxnet-ci] branch jenkins-run-statistics created (now b7c72de)
This is an automated email from the ASF dual-hosted git repository. marcoabreu pushed a change to branch jenkins-run-statistics in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-ci.git. at b7c72de Add jenkins run statistics This branch includes the following new commits: new b7c72de Add jenkins run statistics The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'.
ChaiBapchya commented on issue #15882: Move Windows CI build to a 64-bit toolchain to fix 'out of heap space'. URL: https://github.com/apache/incubator-mxnet/pull/15882#issuecomment-521802937 Yes please. I've been retriggering my 5 PRs for last 2 days (atleast 2-3 times per PR) Would be great if we address this time-out issue! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
