Neutron3529 edited a comment on pull request #19748:
URL: https://github.com/apache/incubator-mxnet/pull/19748#issuecomment-761559675
> What's the advantage of this PR compared to use explicit
`iter(train_iter)` in the `auto_reload=False` case? As your example code shows,
the `auto_reload=True` case makes it much harder to understand what is going on
with the Dataloader, especially if users create multiple `iters` of the same
Dataloader at the same time.. Without your PR, user can just call `iter`
multiple times explicitly and obtain the same behavior. "Explicit is better
than implicit." (Zen of Python)
if the users create multiple iters of the same Dataloader at the same time,
with this PR, user can also call `iter` multiple times.
the first time, Dataloader generate the prefetched iter,
and the rest of the time, dataloader return an iter and start to fetch data
for the iter.
if you do not modify the Dataloader(e.g., modify `self._dataset`), all the
result except the performance are the same for the iters.
The reason why using `Dataloader` with `auto_reload` is:
1. MXNet 2.0's `DataLoader` with the default `nopython` mode prefetch data
by default.
2. There is only one `iter` for a `DataLoader` in most of the cases.(Thus
only one prefetched `iter` is generated.)
3. if we call `iter` explicitly, we should call it twice (one right after
the define of the `DataLoader`, and another one after the previous iter is
consumed).
4. (maybe we should not using `with ag.record():` since "Explicit is better
than implicit." (Zen of Python))
Actually, I do not think there is something that was not considered in my
Example, I created 4 iters, the only differences among them is the prefetching
behavior:
```python
>>> from mxnet.gluon.data import DataLoader, ArrayDataset
>>> train_data = ArrayDataset([i for i in range(10)],[9-i for i in
range(10)])
>>> def transform_train(sample):
... if sample == 0 : print('(pre)fetching data here')
... return sample
...
>>> train_iter = DataLoader(train_data.transform_first(transform_train),
... auto_reload=False,
batch_size=1,num_workers=1)
>>> # no prefetch is performed, the prefetch & autoload start after
>>> # train_iter.__iter__() is called.
>>> for i in train_iter:pass
(pre)fetching data here
>>> train_iter = DataLoader(train_data.transform_first(transform_train),
... batch_size=1,num_workers=1)
(pre)fetching data here
>>> it = iter(train_iter) # nothing is generated since lazy-evaluation
occurs
>>> it2 = iter(train_iter)
>>> it3 = iter(train_iter)
>>> it4 = iter(train_iter)
>>> _ = next(it2) # the first iter we are using is the prefetched iter.
>>> _ = next(it) # since the prefetched iter is cconsumed, we have to
fetch data for `it`.
(pre)fetching data here
>>> _ = [None for _ in it3]
(pre)fetching data here
(pre)fetching data here
>>> # Here, 2 prefetches are triggered, one is fetching the first batch
of `it3` and
>>> # another is when `it3` yield its last item, a prefetch is
automatically performed.
>>> _ = [None for _ in it]
>>> # no prefetch is happened since train_loader has already prefetch
data.
>>> _ = next(it4)
>>> # since the prefetch is performed, it4 become the prefetched iter.
>>>
>>> test_data = ArrayDataset([i for i in range(10)],[9-i for i in
range(10)])
>>> test_iter = PrefetchedDataLoader(test_data,
... batch_size=1,num_workers=1)
>>> for epoch in range(200):
... # there is almost no difference between it and the default
DataLoader
... for data, label in train_iter:
... # training...
... for data, label in test_iter:
... # testing...
```
What's more, the `# nothing is generated since lazy-evaluation occurs`
comment is a feature, not a bug.
Although we could using something like `it5 = train_iter._prefetch_iter()`
forcing the dataloader generate a prefetched iter, users may not using such
technique since my test results shows it is a little bit slower open up several
prefetched iter at once.
`cost=115.23376344s` here, may be `cost=129.67192333600002s` if we are using
the old `Dataloader`.
but if we start a second prefetched `iter`(see #19738 ), it would be
slower(`cost=125.498369854s`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
