smukund28 opened a new issue #16096: FixedBucketSampler needs last_batch = discard feature URL: https://github.com/apache/incubator-mxnet/issues/16096 Feature Request : When DataLoader uses a Sampler, one cannot rely on the last_batch='discard' feature. It would be nice to have this in place for Sampler as well. What have you tried to solve it? One way to overcome this error is to pad the samples and load the data. Another way is to simply check for the batch size in the loop and discard if the sample size does not match the batch size.. Example: for batch_id, (token_ids, valid_length, segment_ids, label) in enumerate(bert_dataloader): if len(token_ids) != batch_size: continue token_ids1 = gluon.utils.split_and_load(token_ids, ctx, even_split=False ) valid_length1 = gluon.utils.split_and_load(valid_length, ctx, even_split=False ) segment_ids1 = gluon.utils.split_and_load(segment_ids, ctx, even_split=False) label1 = gluon.utils.split_and_load(label, ctx, even_split=False)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
