1996fanrui commented on pull request #13885: URL: https://github.com/apache/flink/pull/13885#issuecomment-732264590
> Hi @1996fanrui , > > that's an interesting discovery and investigation that you did there! > > I think the approach on the filesystem level is also much better than the previous way. Let's try to not change any public API (FileSystem) as this would slow down the progress. > > I'd probably focus on Hadoop file systems entirely (for now). What I'd propose is the following: > > * Use `HadoopFsFactory#configure` to extract the buffer size and pass it to the `ctor` of all filesystems created by the factory. > * Use that default buffer size in `HadoopFileSystem#open(Path)` to call `#open(Path, int)`. > * `HadoopFileSystem#open(Path, int)` should use the buffer size both in the call to Hadoop and to wrap it as you did as in the `BufferedFSInputStream`. I dug a bit into the Hadoop code and noticed that the cache is by default just 4kb. So even if we have cache on top of it with 64kb, we would still need to ask Hadoop several times. > > So, that means you are not adding any new methods, but just modify existing ones. Hi @AHeise , thanks for your suggestions. In the process of modifying the code, I also felt that the public API was modified. There may be some unknown effects. I have some questions now: This feature can optimize the IO performance of `LocalFileSystem`, so can it be applied to `LocalFileSystem` and `HadoopFileSystem`? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
