advancedxy commented on a change in pull request #30728:
URL: https://github.com/apache/spark/pull/30728#discussion_r543434196
##########
File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
##########
@@ -300,7 +300,8 @@ private void advanceToNextPage() {
@Override
public boolean hasNext() {
if (numRecords == 0) {
- if (reader != null) {
+ // hasNext maybe called multiple times. Let's guard that
+ if (reader != null && spillWriters.size() > 0) {
Review comment:
Yeah, I thought about that. It's weird to do the check here. The better
solution would be add a new field to indicate the iterator has reached to its
end, but seems quite overkill.
How do you like the following?
```
private void handleFailedDelete() {
if (spillWriters.size() > 0) {
// remove the spill file from disk
File file = spillWriters.removeFirst().getFile();
if (file != null && file.exists() && !file.delete()) {
logger.error("Was unable to delete spill file {}",
file.getAbsolutePath());
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
