Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11628#issuecomment-196913296
Ah OK, so the test already correctly made sure the exception is thrown.
That's fine then. I personally expect to see checking `hasNext` in `next`, or
equivalent, since the method needs to know when to throw
`NoSuchElementException`. Here I did it to get the cleanup behavior that
`hasNext` already for free.
OK, obviously I like the change as I think you're now fixing several
similar possible problems and tightening up cleanup.
Is this the problem sequence?
- Exception check happens first and sees no exception
- Exception occurs
- `lines.hasNext` reports `false` (? as opposed to throwing an error in
this case? maybe)
- Wait for process to complete
- No exception is returned, and it silently claims there are no more
elements, so `hasNext` isn't checked again
In general it's of course possible for the exception to happen at anytime
and the idea is that the _next_ method invocation finds out about it, so the
first 2 steps are normal. However if the underlying iterator doesn't fail
explicitly I see the problem here.
At the same time it seems weird to proceed to check `lines.hasNext` if we
know there's an error already. Is the right-er fix to also check it if
`lines.hasNext` is false? Also checking at the end unilaterally is OK, maybe
just overkill.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
