Neutron3529 commented on pull request #19738:
URL: https://github.com/apache/incubator-mxnet/pull/19738#issuecomment-757478101
> Have you tried using `iter`? This should allow you to obtain the same
behavior without making a change in the `v1.x` branch
Excellent idea..
maybe what we should do is modify the document/tutorial.
I found using `iter` is faster than using the ugly PR here.
```
import mxnet as mx
from mxnet.gluon.data import DataLoader,ArrayDataset
from time import sleep,perf_counter_ns
train_data=ArrayDataset(mx.nd.array([[i] for i in
range(50000)]),mx.nd.array([[99-i] for i in range(50000)]))
test_data=ArrayDataset(mx.nd.array([[i] for i in
range(10000)]),mx.nd.array([[99-i] for i in range(10000)]))
def transform_train(sample):
sleep(0.0016)
return sample
def transform_test(sample):
sleep(0.0008)
return sample
train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10)
test_iter =DataLoader(test_data .transform_first(transform_test
),batch_size=500,num_workers=10)
itn=iter(train_iter)
itt=iter(test_iter)
if True:
tic=perf_counter_ns()
for epoch in range(10):
print("epoch"+str(epoch)+" start at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
for i in itn:
sleep(0.1)
itn=iter(train_iter)
print(" finished train phase at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
for i in itt:
sleep(0.05)
itt=iter(test_iter)
print(" finished test phase at
"+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
print("cost="+str((perf_counter_ns()-tic)*1e-9)+"s")
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
