[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397986#comment-16397986
 ] 

Chris Olivier commented on MXNET-11:


Yeah great performance improvement!  We definitely have requests to run
parallel inference in the same process. Each one loading the model
separately isn’t a realistic solution. Do you know if other libraries can
do this?





> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org

[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395518#comment-16395518
 ] 

Chris Olivier commented on MXNET-11:


I suppose the multiple threads would call the C API.  Python is a bit tricky to 
get it to do multithreading, so I wouldn't be concerned with a python entry 
point at this point.

So above, you launched many processes, loaded a model and ran inference in 
parallel?

Or you loaded a model in many threads and ran inference through those?

The main problem right now is that there's not a good way to *share* parameters 
between graphs, so that would probably be some large bulk of the work. There's 
actually several other use-cases for this, including Keras integration.

 

 

> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org

[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395506#comment-16395506
 ] 

Chris Olivier commented on MXNET-11:


oops caps...


On Mon, Mar 12, 2018 at 9:56 AM, Chris Olivier 



> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org

[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395505#comment-16395505
 ] 

Chris Olivier commented on MXNET-11:


i CAN;T SEEM TO ADD A COMMENT FOR SOME REASON...

On Sun, Mar 11, 2018 at 10:43 PM, Patric Zhao (JIRA) 



> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org

[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392434#comment-16392434
 ] 

Chris Olivier commented on MXNET-11:


There is also a claim that the same thread must always call into the C API from 
python, but it’s not clear to me why this restriction exists (I haven’t 
personally seen any code that would be sensitive to that)

> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org

[jira] [Commented] (MXNET-11) Multithreaded Inference

[Chris Olivier (JIRA)]

[ 
https://issues.apache.org/jira/browse/MXNET-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392433#comment-16392433
 ] 

Chris Olivier commented on MXNET-11:


Currently, it is not possible to create two graphs which share the same 
parameters (weights, biases, etc) — without creating two distinct copies of the 
parameters — and start running inference on the two graphs in parallel. This is 
a common use-case, especially since for many models, we won’t reach 100% CPU 
utilization on a single graph. 

Currently the only way to accomplish this by forking the process, but that 
approach also has a lot of drawbacks, as you can imagine, as isn’t suitable for 
most use-cases.

> Multithreaded Inference
> ---
>
> Key: MXNET-11
> URL: https://issues.apache.org/jira/browse/MXNET-11
> Project: Apache MXNet
>  Issue Type: Epic
>  Components: MXNet Engine
>Reporter: Chris Olivier
>Priority: Major
>  Labels: inference
>
> Add the ability to do multithreaded inference without using fork() or using 
> multiple copies of a given model



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org
For additional commands, e-mail: issues-h...@mxnet.apache.org