Neutron3529 opened a new pull request #19748:
URL: https://github.com/apache/incubator-mxnet/pull/19748
Now, since my programming skill is very poor, this `PrefetchedDataLoader`
only allow generate a single iter at the same time.
the benefit of `PrefetchedDataLoader` is that, `PrefetchedDataLoader`
provides better performance with a simple replacement in most of the existing
codes.
test:
```python
$ cat iternew.py && python iternew.py
import mxnet as mx
from mxnet.gluon.data import PrefetchedDataLoader as DataLoader,ArrayDataset
from time import sleep,perf_counter_ns
train_data=ArrayDataset(mx.nd.array([[i] for i in
range(50000)]),mx.nd.array([[99-i] for i in range(50000)]))
test_data=ArrayDataset(mx.nd.array([[i] for i in
range(10000)]),mx.nd.array([[99-i] for i in range(10000)]))
def transform_train(sample):
sleep(0.0016)
return sample
def transform_test(sample):
sleep(0.0008)
return sample
train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10)
test_iter =DataLoader(test_data .transform_first(transform_test
),batch_size=500,num_workers=10)
if True:
tic=perf_counter_ns()
for epoch in range(10):
print("epoch"+str(epoch)+" start at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
for i in train_iter:
sleep(0.1)
print(" finished train phase at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
for i in test_iter:
sleep(0.05)
print(" finished test phase at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
print("cost="+str((perf_counter_ns()-tic)*1e-9)+"s")
epoch0 start at 0.0s
finished train phase at 11.25s
finished test phase at 12.31s
epoch1 start at 12.31s
finished train phase at 22.62s
finished test phase at 23.68s
epoch2 start at 23.68s
finished train phase at 34.03s
finished test phase at 35.09s
epoch3 start at 35.09s
finished train phase at 45.41s
finished test phase at 46.48s
epoch4 start at 46.48s
finished train phase at 56.82s
finished test phase at 57.88s
epoch5 start at 57.88s
finished train phase at 68.24s
finished test phase at 69.3s
epoch6 start at 69.3s
finished train phase at 79.65s
finished test phase at 80.71s
epoch7 start at 80.71s
finished train phase at 91.04s
finished test phase at 92.11s
epoch8 start at 92.11s
finished train phase at 102.46s
finished test phase at 103.53s
epoch9 start at 103.53s
finished train phase at 113.89s
finished test phase at 114.95s
cost=114.94954171600001s
```
(cost ~`129.67192333600002s` if we are using `Dataloader` rather than
`PrefetchedDataLoader`)
## Description ##
(Brief description on what this PR is about)
## Checklist ##
### Essentials ###
- [ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL],
[FEATURE], [DOC], etc)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage
- [ ] Code is well-documented
### Changes ###
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
## Comments ##
- If this change is a backward incompatible change, why must this change be
made.
- Interesting edge cases to note here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
