vrozov commented on issue #272: Refactor AvroIterable and prevent multiple iterators URL: https://github.com/apache/incubator-iceberg/pull/272#issuecomment-510652572 @rdblue Please take a look at `DataFileReader`. It supports only one iterator. `AvroIterable` also supports only one iterator and it provides the signal (`IllegalStateException`) should somebody try to obtain a second iterator. There is a difference between iterating over in-memory collection and iterating over records in file: list, map, set iterators do not need to be closed and there is no significant cost associated with the second iteration over in-memory collection. The point of using Iterable even with a single iterator() is 1. support "for-each loop" 2. ability to pass Iterable and delay creating Iterator instance. I considered the approach to close an iterator when `hasNext()` returns `false`, but rejected it as it is not necessary to exhaust iteration (even though most of the time iterator will be fully exhausted), so the resource may be left open. The PR does not fix any existing resource leakage or shortage. The goal of the PR is to prevent such a possibility for future modifications.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
