[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-330128520 @szha Hello, I've just created a pull request. What I did: 1. Add transform utils to gluon.data. 2. Add datasets to gluon.data.vison What are left: 1. An efficient multi-process loader I notice there is already one single process loader. `MXBox` has a multi-thread loader, however due to the GIL of python, multi-thread does not help much when loading large image. Meanwhile, fetching large data from queue will cause large overhead in multi-process. The best solution I have seen so far is the pin_memory in pytorch.Tensor -- which passes a reference between processes instead of copying. 2. More dataesets like SVHN / COCO etc 3. New feature `gpuManager` `MXNet` does not have API for querying available gpu devices and usage status of each gpu. Hence I add `gpuManager` to `MXBox`. With this feature, we no longer need to specify devices manually, it can automatically find free GPUs for your program. ```python # apply 3 `free` gpus on this server # `Free` : Memory usage percentage is under 20% gpus = gpuManager.allocate( 3, memory_limit=0.2) # return a list of available gpus ``` I just added it `MXBox`, how do you think of merging it to `MXNet`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-330128520 @szha Hello, I've just created a pull request. What I did: 1. Add transform utils to gluon.data. 2. Add datasets to gluon.data.vison What are left: 1. An efficient multi-process loader I notice there is already one single process loader. `MXBox` has a multi-thread loader, however due to the GIL of python, multi-thread does not help much when loading large image. Meanwhile, fetching large data from queue will cause large overhead in multi-process. The best solution I have seen so far is the pin_memory in pytorch.Tensor -- which passes a reference between processes instead of copying. 2. More dataesets like SVHN / COCO etc 3. New feature `gpuManager` Without specifying devices manually, this API can automatically find free GPUs for your program. ```python # apply 3 `free` gpus on this server # `Free` : Memory usage percentage is under 20% gpus = gpuManager.allocate( 3, memory_limit=0.2) # return a list of available gpus ``` I just added it `MXBox`, how do you think of merging it? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-323653480 Hi there, I am merging MXbox to gluon.data. Which branch should I create pull request to? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-321681785 @szha Gluon's documentation seems to be under construction. Could u show some examples of `mx.symbol`? In fact `mxbox` provides [`mx.transforms.lambda`](https://github.com/Lyken17/mxbox/blob/master/mxbox/transforms/general.py#L140) which allows user to easily create extensions . Thus supports for `mx.symbol` should not be a trouble issue. @zhreshold Gluon and mxbox are both inspired by torchvision, so they have many same functions. What I think is to keep the relation of `mxbox` and `mxnet` like `pytorch` and `torchvision` -- standalone but strongly connected. The advantage of doing this is people can develop `mxbox` without affecting `mxnet` (push to `mxbox` only), and user can always enjoy the latest update of mxbox from PyPi. Maintaining a standalone toolbox will not be a nightmare, instead, it brings faster update -- for example, a new network comes out, once I get the definition scripts and the pretrained weight, I can upload to `mxnet` and give a release immediately, where in mxnet, it requires a reviewer to merge in and extra time to release a new version on PyPi. We all aim for a good toolbox, and I think a standalone repo is a better choice. But as I mentioned, my free time on next semester is very limited, so to merge in or not, actually depends on later maintainers' idea. If he thinks merging to be a better choice, I will create a pull request after construction for `model` is finished. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-321681785 @szha Gluon's documentation seems to be under construction. Could u show some examples of `mx.symbol`? In fact `mxbox` provides [`mx.transforms.lambda`](https://github.com/Lyken17/mxbox/blob/master/mxbox/transforms/general.py#L140) which allows user to easily create extensions . Thus supports for `mx.symbol` should not be a trouble issue. @zhreshold Gluon and mxbox are both inspired by torchvision, so they have many same functions. What I think is to keep the relation of `mxbox` and `mxnet` like `pytorch` and `torchvision` -- standalone but strongly connected. The advantage of doing this is people can develop `mxbox` without affecting `mxnet` (push to `mxbox` only), and user can always enjoy the latest update of mxbox from PyPi. Maintaining a standalone toolbox will not be a nightmare, instead, it will brings faster update -- for example, a new network comes out, I can upload to `mxnet` and give a release immediately, once I get the definition scripts and the pretrained weight. where in mxnet, it requires a reviewer to merge in and extra time for a new release on PyPi. We all aim for a good toolbox, and as I mentioned, my free time on next semester is very limited. So to merge in or not, actually depends on later maintainers' idea. If you all think merging to be a better choice, I will create a pull request after construction for `model` is finished. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework.
Lyken17 commented on issue #7376: MXbox -- a simple and flexible vision toolbox for mxnet framework. URL: https://github.com/apache/incubator-mxnet/issues/7376#issuecomment-321065474 I am going to apply for graduate schools this fall, so the time I can spend on mxbox will be very limited. I think moving mxbox to community would be a good idea. And also, as for pretrained models, I do need help from community. But I am thinking, is it better to merge mxbox into gluon.data or keep it as a standalone tools, like the ps-lite and tensorboard in dmlc. Since it is not strictly binded with gluon/ndarray, keeping it standalone can make iterations of develop faster (without changes to mxnet main branch). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
