[GitHub] mwbyeon commented on issue #9203: compile error in plugin opencv
mwbyeon commented on issue #9203: compile error in plugin opencv
URL:
https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354069809
@TX2012LH
Sorry. I missed that.
Please check below.
1. package path of mxnet
```bash
$ python
>>> import mxnet
>>> mxnet.base._LIB
>>>
```
2. show functions using `nm` command
```bash
$ nm -gC
{HIDDEN}/python3.6/site-packages/mxnet-1.0.1-py3.6.egg/mxnet/libmxnet.so |
grep MXCV
032ec150 T MXCVResize(void*, unsigned int, unsigned int, int, void**)
032ea2e0 T MXCVImdecode(unsigned char const*, unsigned int, int,
void**)
032e7170 T MXCVcopyMakeBorder(void*, int, int, int, int, int,
double, void**)
```
if `MXCV*` functions not exists, rebuild and reinstall mxnet.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] fcbruce commented on issue #9207: It seems nd.array or NDArrayIter not support too large array
fcbruce commented on issue #9207: It seems nd.array or NDArrayIter not support too large array URL: https://github.com/apache/incubator-mxnet/issues/9207#issuecomment-354069172 Sorry, the version on CentOS is 1.0.0 and on MacOS is 0.11.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] fcbruce opened a new issue #9207: It seems nd.array or NDArrayIter not support too large array
fcbruce opened a new issue #9207: It seems nd.array or NDArrayIter not support too large array URL: https://github.com/apache/incubator-mxnet/issues/9207 ## Description I have a large array, cannot be transformed into `nd.array` ## Environment info (Required) CentOS and MacOS Package used (Python/R/Scala/Julia): Python ## Build info (Required if built from source) install by pip, mxnet=1.0.0 ## Error Message: ``` 15:31:56] /Users/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:308: [15:31:56] include/mxnet/./tensor_blob.h:275: Check failed: this->shape_.Size() == shape.Size() (655360 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements Stack trace returned 7 entries: [bt] (0) 0 libmxnet.so 0x000d378eaad8 _ZN4dmlc15LogMessageFatalD2Ev + 40 [bt] (1) 1 libmxnet.so 0x000d3790cae9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777 [bt] (2) 2 libmxnet.so 0x000d380fa0be _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382 [bt] (3) 3 libmxnet.so 0x000d380d9673 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1139 [bt] (4) 4 libmxnet.so 0x000d37fcc1fd MXNDArraySyncCopyFromCPU + 13 [bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x000101eb742f ffi_call_unix64 + 79 [bt] (6) 6 ??? 0x7fff5e83a820 0x0 + 140734779074592 Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 1295, in array arr[:] = source_array File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 386, in __setitem__ self._sync_copyfrom(value) File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 560, in _sync_copyfrom ctypes.c_size_t(source_array.size))) File "/usr/local/lib/python3.6/site-packages/mxnet/base.py", line 129, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: [15:31:56] include/mxnet/./tensor_blob.h:275: Check failed: this->shape_.Size() == shape.Size() (655360 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements Stack trace returned 7 entries: [bt] (0) 0 libmxnet.so 0x000d378eaad8 _ZN4dmlc15LogMessageFatalD2Ev + 40 [bt] (1) 1 libmxnet.so 0x000d3790cae9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777 [bt] (2) 2 libmxnet.so 0x000d380fa0be _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382 [bt] (3) 3 libmxnet.so 0x000d380d9673 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1139 [bt] (4) 4 libmxnet.so 0x000d37fcc1fd MXNDArraySyncCopyFromCPU + 13 [bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x000101eb742f ffi_call_unix64 + 79 [bt] (6) 6 ??? 0x7fff5e83a820 0x0 + 140734779074592 ``` ## Minimum reproducible example ``` import numpy as np import mxnet as mx X = np.zeros((2, 32768), dtypes=np.float32) mx.nd.array(x) ``` ## Steps to reproduce just run the code above This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? ``` python import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ``` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? ``` python import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ``` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? `import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100))` ` ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769514 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @eric-haibin-lin Yes, we need to construct the table every time. Our current implementation is to first construct a cumsum table and then search the table to get the result. The construction complexity of the current implementation is O(N) and the best possible search complexity if O(log(N)). For this approach, the construction complexity is still O(N) and the search complexity is O(1), which will be faster when we draw multiple samples from the same multinomial distribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769514 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @eric-haibin-lin Yes, we need to construct the table every time. Our current implementation is to first construct a cumsum table and then search the table to get the result. The construction complexity of the current implementation is O(N) and the best possible search complexity if O(log(N)). For this approach, the construction complexity is still O(N) and the search complexity is O(1), which will faster when we draw multiple samples from the same multinomial distribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769514 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @eric-haibin-lin Yes, we need to construct the table every time. Our current implementation is to first construct a cumsum table and then search the table to get the result. The construction complexity of the current implementation is O(N) and the best possible search complexity if O(log(N)). For this approach, the construction complexity is O(N) and the search complexity is O(1), which will faster when we draw multiple samples from the same multinomial distribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? ` import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? `import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100))` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769514 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @eric-haibin-lin Yes, we need to construct the table every time. Our current implementation is to first construct a cumsum table and then search the table to get the result. The search complexity of the current implementation is O(N) and the best possible search complexity if O(log(N)). For this approach, the construction complexity is O(N) and the search complexity is O(1), which will faster when we draw multiple samples from the same multinomial distribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? ` import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? ` import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? `import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it as follows. What may be the problem ? `import opencv import numpy as np a = np.random.randint(100, size=(30,30,3)) b = opencv.resize(a, (100, 100)) ` @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769514 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @eric-haibin-lin Yes, we need to construct the table every time. Our current implementation is to first construct a cumsum table and then search the columns to get the result. The search complexity of the current implementation is O(N) and the best possible search complexity if O(log(N)). For this approach, the construction complexity is O(N) and the search complexity is O(1), which will faster when we draw multiple samples from the same multinomial distribution. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully, `undefined symbol _LIB.MXCVResize` appears when call it. What may be the problem ? @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354062727 Although it is compiled successfully after fixing this bug, `undefined symbol _LIB.MXCVResize` appears when call it. What may be the problem ? @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mwbyeon commented on issue #9203: compile error in plugin opencv
mwbyeon commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354061951 @TX2012LH Thanks. It's a bug in `cv_api.cc`. I created a new PR(#9206) to fix this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158769061 ## File path: example/gluon/sampler/alias_method.py ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# coding: utf-8 +# pylint: skip-file + +import numpy as np +import numpy.random as npr +from collections import Counter + +import mxnet.gluon as gluon + +K = 5 +N = 1000 + +# Get a random probability vector. +probs = npr.dirichlet(np.ones(K), 1).ravel() + +# Construct the table. +alias_method_sampler = gluon.data.AliasMethodSampler(K, probs) + +# Generate variates. +X = alias_method_sampler.draw(N).asnumpy() +c = Counter(X) + +# check sampled probabilities +sampled_probs = [float(x)/N for x in c.values()] Review comment: @zihaolucky I think it could also call the chi-square test in https://github.com/apache/incubator-mxnet/pull/9129 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mwbyeon commented on issue #9203: compile error in plugin opencv
mwbyeon commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354061951 @TX2012LH Thanks. It's a bug in `cv_api.cc`. I created a new PR(#9206) to fix this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mwbyeon opened a new pull request #9206: Fix incude error in plugin/opencv/cv_api.cc
mwbyeon opened a new pull request #9206: Fix incude error in plugin/opencv/cv_api.cc URL: https://github.com/apache/incubator-mxnet/pull/9206 ## Description ## * Fix include error in `plugin/opencv/cv_api.cc` (related #9203) * `c_api_error.h` does not exists. it replaced to `c_api_common.h`. ## Checklist ## ### Essentials ### - [x] Passed code style checking (`make lint`) - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage: - Unit tests are added for small changes to verify correctness (e.g. adding a new operator) - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore) - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL) - [x] Code is well-documented: - For user-facing API changes, API doc string has been updated. - For new C++ functions in header files, their functionalities and arguments are documented. - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [x] fix include path This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] vsooda closed pull request #8950: fix build error in clang-900.0.38 in mac
vsooda closed pull request #8950: fix build error in clang-900.0.38 in mac
URL: https://github.com/apache/incubator-mxnet/pull/8950
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/operator/l2_normalization-inl.h
b/src/operator/l2_normalization-inl.h
index cb8e740d7f..fd38dafadb 100644
--- a/src/operator/l2_normalization-inl.h
+++ b/src/operator/l2_normalization-inl.h
@@ -96,7 +96,7 @@ class L2NormalizationOp : public Operator {
Tensor norm = out_data[l2_normalization::kNorm].get(s);
norm = sumall_except_dim<0>(F(data));
norm = F(norm + param_.eps);
- out = data / broadcast<0>(norm, out.shape_);
+ out = data / mshadow::expr::broadcast<0>(norm, out.shape_);
} else if (param_.mode == l2_normalization::kChannel) {
CHECK_GE(orig_shape.ndim(), 3U);
Shape<3> dshape = Shape3(orig_shape[0], orig_shape[1],
@@ -159,8 +159,8 @@ class L2NormalizationOp : public Operator {
.get_space(mshadow::Shape1(data.shape_[0]), s);
temp = sumall_except_dim<0>(grad_out * data);
Assign(grad_in, req[l2_normalization::kData],
-(grad_out - data * broadcast<0>(temp, data.shape_)) /
-broadcast<0>(norm, data.shape_));
+(grad_out - data * mshadow::expr::broadcast<0>(temp, data.shape_)) /
+mshadow::expr::broadcast<0>(norm, data.shape_));
} else if (param_.mode == l2_normalization::kChannel) {
CHECK_GE(orig_shape.ndim(), 3U);
Shape<3> dshape = Shape3(orig_shape[0], orig_shape[1],
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] vsooda commented on issue #9060: Remove defunct demo
vsooda commented on issue #9060: Remove defunct demo URL: https://github.com/apache/incubator-mxnet/pull/9060#issuecomment-354058923 @piiswrong @simoncorstonoliver I had make this work in February. what is the problem you must remove this example. just because you could not let it work? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] formath closed issue #9023: Multiple featureid in one field in wide example
formath closed issue #9023: Multiple featureid in one field in wide example URL: https://github.com/apache/incubator-mxnet/issues/9023 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon
eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon
URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158765697
##
File path: python/mxnet/gluon/data/sampler.py
##
@@ -136,3 +138,74 @@ def __len__(self):
raise ValueError(
"last_batch must be one of 'keep', 'discard', or 'rollover', " \
"but got %s"%self._last_batch)
+
+
+class AliasMethodSampler(object):
+""" The Alias Method: Efficient Sampling with Many Discrete Outcomes.
+Can be use in NCELoss.
+
+Parameters
+--
+K : int
+Number of events.
+probs : array
+Probability of each events, corresponds to K.
+
+References
+---
+
https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/
+"""
+def __init__(self, K, probs):
+if K != len(probs):
+raise ValueError("K should be equal to len(probs). K:%d,
len(probs):%d" % (K, len(probs)))
+self.K = K
+self.prob = nd.zeros(K)
+self.alias = nd.zeros(K, dtype='int32')
+
+# Sort the data into the outcomes with probabilities
+# that are larger and smaller than 1/K.
+smaller = []
+larger = []
+for kk, prob in enumerate(probs):
+self.prob[kk] = K*prob
+if self.prob[kk] < 1.0:
Review comment:
Is `asscalar()` missing here? Is using NDArray actually faster thank numpy
array? This comparison converts a ndarray to a scalar. In my experience using a
python list is fastest, followed by numpy array, followed by ndarray. We can
convert the list to mxnet ndarray after the O(N) preprocessing.
You can try a distribution with 1M classes to compare the speed.
@zihaolucky @szha
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon
eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158765827 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: @sxjscience do we expect to construct the table every time when `SampleMultinomial` is called? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon
eric-haibin-lin commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158765522 ## File path: example/gluon/sampler/alias_method.py ## @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# coding: utf-8 +# pylint: skip-file + +import numpy as np +import numpy.random as npr +from collections import Counter + +import mxnet.gluon as gluon + +K = 5 +N = 1000 + +# Get a random probability vector. +probs = npr.dirichlet(np.ones(K), 1).ravel() + +# Construct the table. +alias_method_sampler = gluon.data.AliasMethodSampler(K, probs) + +# Generate variates. +X = alias_method_sampler.draw(N).asnumpy() +c = Counter(X) + +# check sampled probabilities +sampled_probs = [float(x)/N for x in c.values()] Review comment: Will a unit test be added, too? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354054753 ## Description `undefined symbol MXCVResize` after compiling plugin opencv. ## Environment info ubuntu 14.04 mxnet v0.12.0 cuda 8.0+cudnn6.0 build from source ## Build config: USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_OPENCV=1 MXNET_PLUGINS+=plugin/opencv/opencv.mk ## Error Message: No such file `src/c_api/c_api_error.h` included in `cv_api.cc`. I find there is no `c_api_error.h` in directory `src/c_api` and I replace '#include "../../src/c_api/c_api_error.h"` with `#include "../../src/c_api/c_api-common.h"`. It is resolved and can be compiled successfully. But now when I call the `resize` function in `plugin/opencv/opencv.py`, `undefined symbol _LIB.MXCVResize` appears. @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354054753 ## Description `undefined symbol MXCVResize` after compiling plugin opencv. ## Environment info ubuntu 14.04 mxnet v0.12.0 cuda 8.0+cudnn6.0 build from source ## Build config: USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_OPENCV=1 MXNET_PLUGINS+=plugin/opencv/opencv.mk ## Error Message: No such file `src/c_api/c_api_error.h` included in `cv_api.cc`. I find there is no `c_api_error.h` in directory `src/c_api` and I replace '#include "../../src/c_api/c_api_error.h"` with `#include "../../src/c_api/c_api-common.h"`. It is resolved and can be compiled successfully. But now when I call the `resize` function in `plugin/opencv/opencv.py`, `undefined symbol _LIB.MXCVResize' appears. @mwbyeon This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH commented on issue #9203: compile error in plugin opencv
TX2012LH commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354054753 ## Description `undefined symbol MXCVResize` after compiling plugin opencv. ## Environment info ubuntu 14.04 mxnet v0.12.0 cuda 8.0+cudnn6.0 build from source ## Build config: USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_OPENCV=1 MXNET_PLUGINS+=plugin/opencv/opencv.mk ## Error Message: No such file `src/c_api/c_api_error.h` included in `cv_api.cc`. I find there is no `c_api_error.h` in directory `src/c_api` and I replace '#include "../../src/c_api/c_api_error.h"` with `#include "../../src/c_api/c_api-common.h"`. It is resolved and can be compiled successfully. But now when I call the `resize` function in `plugin/opencv/opencv.py`, `undefined symbol _LIB.MXCVResize' appears. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mwbyeon commented on issue #9203: compile error in plugin opencv
mwbyeon commented on issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203#issuecomment-354050412 @TX2012LH please specify more information. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zihaolucky commented on a change in pull request #9195: [WIP]NCE loss gluon
zihaolucky commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158759782 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: Great. I'll try to do this once we finish this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zihaolucky commented on a change in pull request #9195: [WIP]NCE loss gluon
zihaolucky commented on a change in pull request #9195: [WIP]NCE loss gluon
URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158759746
##
File path: python/mxnet/gluon/data/sampler.py
##
@@ -136,3 +138,74 @@ def __len__(self):
raise ValueError(
"last_batch must be one of 'keep', 'discard', or 'rollover', " \
"but got %s"%self._last_batch)
+
+
+class AliasMethodSampler(object):
+""" The Alias Method: Efficient Sampling with Many Discrete Outcomes.
+Can be use in NCELoss.
+
+Parameters
+--
+K : int
+Number of events.
+probs : array
+Probability of each events, corresponds to K.
+
+References
+---
+
https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/
+"""
+def __init__(self, K, probs):
+if K != len(probs):
+raise ValueError("K should be equal to len(probs). K:%d,
len(probs):%d" % (K, len(probs)))
+self.K = K
+self.prob = nd.zeros(K)
+self.alias = nd.zeros(K, dtype='int32')
+
+# Sort the data into the outcomes with probabilities
+# that are larger and smaller than 1/K.
+smaller = []
+larger = []
+for kk, prob in enumerate(probs):
+self.prob[kk] = K*prob
+if self.prob[kk] < 1.0:
+smaller.append(kk)
+else:
+larger.append(kk)
+
+# Loop though and create little binary mixtures that
+# appropriately allocate the larger outcomes over the
+# overall uniform mixture.
+while len(smaller) > 0 and len(larger) > 0:
+small = smaller.pop()
+large = larger.pop()
+
+self.alias[small] = large
+self.prob[large] = (self.prob[large] - 1.0) + self.prob[small]
+
+if self.prob[large] < 1.0:
+smaller.append(large)
+else:
+larger.append(large)
+
+for last_one in smaller+larger:
+self.prob[last_one] = 1
+
+def draw(self, n):
+"""Draw N samples from multinomial
+"""
+samples = nd.zeros(n, dtype='int32')
+
+kk = nd.floor(nd.random.uniform(0, self.K, shape=n), dtype='int32')
+rand = nd.random.uniform(shape=n)
+
+prob = self.prob[kk]
+alias = self.alias[kk]
+
+for i in xrange(n):
Review comment:
Thanks, I would change it later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] chowkamlee81 commented on issue #9179: Sample example codes to work on ConvLSTM and ConvGRU based on 2D images..
chowkamlee81 commented on issue #9179: Sample example codes to work on ConvLSTM and ConvGRU based on 2D images.. URL: https://github.com/apache/incubator-mxnet/issues/9179#issuecomment-354046818 Thanks for your kind reply and necessary inputs. I will try go thru the examples as mentioned in HKO-7 project. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CoinCheung opened a new issue #9205: compile error with clang
CoinCheung opened a new issue #9205: compile error with clang
URL: https://github.com/apache/incubator-mxnet/issues/9205
Note: Providing complete information in the most concise form is the best
way to get help. This issue template serves as the checklist for essential
information to most of the technical issues and bug reports. For non-technical
issues and feature requests, feel free to present the information in what you
believe is the best form.
For Q & A and discussion, please start a discussion thread at
https://discuss.mxnet.io
## Description
I can compile successfully with g++ but error appears with clang.
## Environment info (Required)
```
--Python Info--
Version : 3.6.3
Compiler : GCC 7.2.0
Build: ('default', 'Oct 24 2017 14:48:20')
Arch : ('64bit', '')
Pip Info---
Version : 9.0.1
Directory: /usr/lib/python3.6/site-packages/pip
--MXNet Info---
No MXNet installed.
--System Info--
Platform : Linux-4.13.12-1-ARCH-x86_64-with-arch
system : Linux
node : Arch-R720
release : 4.13.12-1-ARCH
version : #1 SMP PREEMPT Wed Nov 8 11:54:06 CET 2017
--Hardware Info--
machine : x86_64
processor:
Architecture:x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s):1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz
Stepping:9
CPU MHz: 3283.002
CPU max MHz: 3500.
CPU min MHz: 800.
BogoMIPS:4993.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache:256K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64
monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm
3dnowprefetch cpuid_fault intel_pt tpr_shadow vnmi flexpriority ept vpid
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap
clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp
hwp_notify hwp_act_window hwp_epp
--Network Test--
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0252
sec, LOAD: 1.2449 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1978 sec, LOAD:
6.1024 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 1.0039 sec, LOAD:
9.4451 sec.
Timing for FashionMNIST:
https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz,
DNS: 0.1305 sec, LOAD: 1.4144 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0125 sec, LOAD:
7.6978 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0104 sec,
LOAD: 0.6309 sec.
```
Package used (Python/R/Scala/Julia):
(I'm using c++)
For Scala user, please provide:
1. Java version: (`java -version`)
2. Maven version: (`mvn -version`)
3. Scala runtime if applicable: (`scala -version`)
For R user, please provide R `sessionInfo()`:
## Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
clang
MXNet commit hash:
17427d1dc949cb921a190151bc8de74ed50504ef
Build config:
```
export CC = gcc
export CXX = clang++
export NVCC = nvcc
# whether compile with options for MXNet developer
DEV = 0
# whether compile with debug
DEBUG = 0
# whether compile with profiler
USE_PROFILER =
# whether to turn on segfault signal handler to log the stack trace
USE_SIGNAL_HANDLER =
# the additional link flags you want to add
ADD_LDFLAGS =
# the additional compile flags you want to add
ADD_CFLAGS =
#-
# matrix computation libraries for CPU/GPU
#-
# whether use CUDA during compile
USE_CUDA = 1
# add the path to CUDA library to link and compile flag
# if you have already add them to environment variable, leave it as NONE
# USE_CUDA_PATH = /usr/local/cuda
USE_CUDA_PATH = /opt/cuda
# whether use CuDNN R3 library
USE_CUDNN = 1
#whether to use NCCL library
USE_NCCL =
[GitHub] jrosebr1 commented on issue #7557: Parsing Training accuracy, Training loss log files to plot Training accuracy, Training loss curves
jrosebr1 commented on issue #7557: Parsing Training accuracy, Training loss log files to plot Training accuracy, Training loss curves URL: https://github.com/apache/incubator-mxnet/issues/7557#issuecomment-354019241 I had the same issue so I created a Python script to parse the logs. Take a look here: https://www.pyimagesearch.com/2017/12/25/plot-accuracy-loss-mxnet/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158736104 ## File path: python/mxnet/gluon/data/sampler.py ## @@ -136,3 +138,74 @@ def __len__(self): raise ValueError( "last_batch must be one of 'keep', 'discard', or 'rollover', " \ "but got %s"%self._last_batch) + + +class AliasMethodSampler(object): +""" The Alias Method: Efficient Sampling with Many Discrete Outcomes. +Can be use in NCELoss. + +Parameters +-- +K : int +Number of events. +probs : array +Probability of each events, corresponds to K. + +References +--- + https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/ +""" Review comment: Since the AliasMethod is faster than the BinarySearch, we can consider to revise the implementation of SampleMultinomial https://github.com/apache/incubator-mxnet/blob/master/src/operator/random/sample_multinomial_op.h#L111-L135. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158735815
##
File path: python/mxnet/gluon/data/sampler.py
##
@@ -136,3 +138,74 @@ def __len__(self):
raise ValueError(
"last_batch must be one of 'keep', 'discard', or 'rollover', " \
"but got %s"%self._last_batch)
+
+
+class AliasMethodSampler(object):
+""" The Alias Method: Efficient Sampling with Many Discrete Outcomes.
+Can be use in NCELoss.
+
+Parameters
+--
+K : int
+Number of events.
+probs : array
+Probability of each events, corresponds to K.
+
+References
+---
+
https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/
+"""
+def __init__(self, K, probs):
+if K != len(probs):
+raise ValueError("K should be equal to len(probs). K:%d,
len(probs):%d" % (K, len(probs)))
Review comment:
Since K is always equal to len(probs), this argument is unnecessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
sxjscience commented on a change in pull request #9195: [WIP]NCE loss gluon
URL: https://github.com/apache/incubator-mxnet/pull/9195#discussion_r158735738
##
File path: python/mxnet/gluon/data/sampler.py
##
@@ -136,3 +138,74 @@ def __len__(self):
raise ValueError(
"last_batch must be one of 'keep', 'discard', or 'rollover', " \
"but got %s"%self._last_batch)
+
+
+class AliasMethodSampler(object):
+""" The Alias Method: Efficient Sampling with Many Discrete Outcomes.
+Can be use in NCELoss.
+
+Parameters
+--
+K : int
+Number of events.
+probs : array
+Probability of each events, corresponds to K.
+
+References
+---
+
https://hips.seas.harvard.edu/blog/2013/03/03/the-alias-method-efficient-sampling-with-many-discrete-outcomes/
+"""
+def __init__(self, K, probs):
+if K != len(probs):
+raise ValueError("K should be equal to len(probs). K:%d,
len(probs):%d" % (K, len(probs)))
+self.K = K
+self.prob = nd.zeros(K)
+self.alias = nd.zeros(K, dtype='int32')
+
+# Sort the data into the outcomes with probabilities
+# that are larger and smaller than 1/K.
+smaller = []
+larger = []
+for kk, prob in enumerate(probs):
+self.prob[kk] = K*prob
+if self.prob[kk] < 1.0:
+smaller.append(kk)
+else:
+larger.append(kk)
+
+# Loop though and create little binary mixtures that
+# appropriately allocate the larger outcomes over the
+# overall uniform mixture.
+while len(smaller) > 0 and len(larger) > 0:
+small = smaller.pop()
+large = larger.pop()
+
+self.alias[small] = large
+self.prob[large] = (self.prob[large] - 1.0) + self.prob[small]
+
+if self.prob[large] < 1.0:
+smaller.append(large)
+else:
+larger.append(large)
+
+for last_one in smaller+larger:
+self.prob[last_one] = 1
+
+def draw(self, n):
+"""Draw N samples from multinomial
+"""
+samples = nd.zeros(n, dtype='int32')
+
+kk = nd.floor(nd.random.uniform(0, self.K, shape=n), dtype='int32')
+rand = nd.random.uniform(shape=n)
+
+prob = self.prob[kk]
+alias = self.alias[kk]
+
+for i in xrange(n):
Review comment:
Should be `range` in order to support Python3
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sxjscience opened a new pull request #9204: Support negative axis in concat
sxjscience opened a new pull request #9204: Support negative axis in concat URL: https://github.com/apache/incubator-mxnet/pull/9204 ## Description ## Fix https://github.com/apache/incubator-mxnet/issues/9175 ## Checklist ## ### Essentials ### - [x] Passed code style checking (`make lint`) - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage: - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [x] negative axis concat, tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sxjscience commented on issue #9179: Sample example codes to work on ConvLSTM and ConvGRU based on 2D images..
sxjscience commented on issue #9179: Sample example codes to work on ConvLSTM and ConvGRU based on 2D images.. URL: https://github.com/apache/incubator-mxnet/issues/9179#issuecomment-353990817 @chowkamlee81 You can try to use the ConvLSTM/ConvRNN implemented in Gluon: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/gluon/contrib/rnn/conv_rnn_cell.py. Also, you can refer to the [HKO-7](https://github.com/sxjscience/HKO-7) project, which is implemented in MXNet. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] TX2012LH opened a new issue #9203: compile error in plugin opencv
TX2012LH opened a new issue #9203: compile error in plugin opencv URL: https://github.com/apache/incubator-mxnet/issues/9203 There is no such file `src/c_api_error.h' when I compiled it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #8040: Bug: Shape of one dimension lead to crash in LogLoss::Update
szha commented on issue #8040: Bug: Shape of one dimension lead to crash in LogLoss::Update URL: https://github.com/apache/incubator-mxnet/issues/8040#issuecomment-353963195 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #7860: mx.nd.Custom bug
szha commented on issue #7860: mx.nd.Custom bug URL: https://github.com/apache/incubator-mxnet/issues/7860#issuecomment-353963192 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #6785: Error importing mxnet in python, installed using pip
szha commented on issue #6785: Error importing mxnet in python, installed using pip URL: https://github.com/apache/incubator-mxnet/issues/6785#issuecomment-353963188 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #8014: cross compile mxnet for android, without using Amalgamation?
szha commented on issue #8014: cross compile mxnet for android, without using Amalgamation? URL: https://github.com/apache/incubator-mxnet/issues/8014#issuecomment-353963197 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #8021: Bug: arg_name "feat" lead to wrong result
szha commented on issue #8021: Bug: arg_name "feat" lead to wrong result URL: https://github.com/apache/incubator-mxnet/issues/8021#issuecomment-353963194 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #7989: Update metric without considering the dataBatch.pad ?
szha commented on issue #7989: Update metric without considering the dataBatch.pad ? URL: https://github.com/apache/incubator-mxnet/issues/7989#issuecomment-353963191 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] szha commented on issue #7973: C++ demo memory release problem
szha commented on issue #7973: C++ demo memory release problem URL: https://github.com/apache/incubator-mxnet/issues/7973#issuecomment-353963187 @apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage. For general "how-to" questions, our [user forum](https://discuss.mxnet.io/) (and [Chinese version](https://discuss.gluon.ai/)) is a good place to get help. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Godricly commented on issue #9163: how to reshape ouput during forward like caffe?
Godricly commented on issue #9163: how to reshape ouput during forward like caffe? URL: https://github.com/apache/incubator-mxnet/issues/9163#issuecomment-353946432 if you are using ndarray, you can reshape it like numpy array. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
