vrozov commented on issue #272: Refactor AvroIterable and prevent multiple iterators URL: https://github.com/apache/incubator-iceberg/pull/272#issuecomment-509728822 The change helps to avoid resource shortage: as it is `AvroIterable` that is `Closeable`, not the `Iterator`, resources acquired in the `iterator()` won't be closed until `AvroIterable` itself is closed and in the case where multiple iterators are obtained from the same `AvroIterable`, there will be multiple file handles/sockets/http connections acquired and pointing to the same file/bucket that can't be closed individually. Note that `org.apache.avro.file.DataFileReader` follows a similar pattern. It is not possible to get multiple iterators from `DataFileReader` as it always returns the same iterator instance (itself). @rdblue What will be a use case were support for multiple iterators is necessary and that can't be covered by creating multiple `AvroIterable` instead?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
