zhreshold commented on a change in pull request #17841:
URL: https://github.com/apache/incubator-mxnet/pull/17841#discussion_r416868103
##########
File path: python/mxnet/gluon/data/dataloader.py
##########
@@ -572,16 +564,26 @@ def default_batchify_fn(data):
unless you are experiencing timeout and you know it's due to slow data
loading.
Sometimes full `shared_memory` will cause all workers to hang and
causes timeout. In these
cases please reduce `num_workers` or increase system `shared_memory`
size instead.
+ try_nopython : bool, default is None
+ Try compile python dataloading pipeline into pure MXNet c++
implementation. The benefit is
+ potentially faster iteration, no `shared_memory` usage, and less
processes managed by python.
+ The compilation is not gauranteed to support all use cases, but it
will fallback to python in
+ case of failure. You can set `try_nopython` to `False` to disable
auto-detection of the
+ compilation feature or leave it to `None` to allow MXNet to determine
it automatically.
+ If you request `try_nopython` to `True` and the compilation fails, it
will raise a warning and
Review comment:
Modified to Raise RuntimeError instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
