Neutron3529 commented on pull request #19738:
URL: https://github.com/apache/incubator-mxnet/pull/19738#issuecomment-757446588
> ```python
> dataset = create_dataset()
> dataloader = DataLoader(dataset)
> for epoch in range(start_epoch, end_epoch):
> for x in dataloader:
> y = model(x)
> dataset.set_epoch(epoch) # change the state of dataset
> ```
Thanks for your reply. Due to lack of knowledge, I don't know how to change
the state of dataset, I wrote some test code and figure out that
`dataset.set_epoch(epoch)` won't affect old `dataloader`
```
import mxnet as mx
from mxnet.gluon.data import DataLoader,ArrayDataset
from time import sleep,perf_counter_ns
def transform_train(sample):
sleep(0.0016)
return sample
train_data=ArrayDataset(mx.nd.array([[i] for i in
range(50000)]),mx.nd.array([[99-i] for i in range(50000)]))
train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10)
train_data[0][0][0]+=2
print(train_data[0])
for i in train_iter:print(i[0][0].asscalar(),i[1][0].asscalar());break
# print `0.0 99.0`, rather than `2.0 99.0`
train_data=ArrayDataset(mx.nd.array([[i] for i in
range(50000)]),mx.nd.array([[99-i] for i in
range(50000)])).transform_first(transform_train)
train_iter=DataLoader(train_data,batch_size=500,num_workers=10)
train_data[0][0][0]+=2
print(train_data[0])
for i in train_iter:print(i[0][0].asscalar(),i[1][0].asscalar());break
# print `0.0 99.0`, rather than `2.0 99.0`
train_iter=DataLoader(train_data,batch_size=500,num_workers=10)
for i in train_iter:print(i[0][0].asscalar(),i[1][0].asscalar());break
# print `2.0 99.0`, which shows that, if we change the old dataset, we must
create a new dataloader, the change does not affect the old iter.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
