bshashikant commented on pull request #1523: URL: https://github.com/apache/ozone/pull/1523#issuecomment-725268132
> Thanks @bshashikant for taking a look. > > > 1. In unbuffer, do we need to remove the corresponding blockInputStreams and chunkInputStreams which are already read? > > I don't think they should be removed. They do not seem to add significant overhead. The stream can be `seek`-ed both forward and backward. > > > 1. While the connection to datanode is closed, connection to OM still is kept open, do we need to close this as well? > > As far as I see, connection to OM is in the FileSystem instance (or rather the `OzoneClientAdapter` implementation), which is longer lived than the InputStreams (and does not support unbuffer). > > > 1. As i see the last position is cached, so after unbuffer is called, it will go into corresponding blockInputStream and chunkInputStream and starts reading again. What if the pipelne is not valid anymore i.e, the datanodes containing the blocks is replicated to a different set of datanodes? Do we need to handle this? > > Ideally we should. But is this really specific to unbuffering? I would think it can happen during a regular read, too. Thanks @adoroszlai . I think with unbufffering , we would need to refresh the pipeline. Usually, while truing to read a block. we first try to read the chunkInfo from datanodes during BlocknInputStream initialize() , and if that fails, we try to refresh the pipeline in case of regular read. With unbuffer, connections to datanodes is closed, and the data will be the probably be read at later point of time. In cases as such, we might need to reinitialize BlockInputStream again. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
