anujmodi2021 commented on PR #6270: URL: https://github.com/apache/hadoop/pull/6270#issuecomment-1875108638
> > I think not only footer reads but this can be expanded to other prefetches as well. > > Especially small files that are read fully can be cached such that multiple streams can be catered to. > > i don't know how common that use is...whereas for spark/tez and workers, reopening the same file is not unusual -they just process different parts. > > I think this is why prefecting doesn't do anything for orc/parquet. Note that impala does cache the column indexes/page indexes so it doesn't need the filesystem to secretly do it for them. Yes, we have also had similar observations. But I feel this cross-stream caching is a good idea for both footer reads, and small files read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
