Hi guys, I think it really depends on the application that you are using. In my case on HDFS, we have quite often thousands of custom applications that read the same data/block at more or less the same time. I can see that it saves us a lot of IOPS because after some time it reads straight from the memory and I see less load on the disk. I don't know how much benefit we get from the caching compared to re-read each time from the disk but I imagine it's not null plus it decreases the load on the disk allowing them to do other stuff. Now you are the expert guys ^^, if you think there are other things to do to improve the performance, then it's clearly not a priority.
I don't know if other people have opinions on it. Best regards, Michel Le jeu. 7 janv. 2021 à 08:38, Wei-Chiu Chuang <weic...@apache.org> a écrit : > To be honest, the HDFS caching didn't take off because somehow it didn't > improve application performance a lot (speaking from an Impala perspective, > I've been told it seems to help with Solr). > > As a matter of fact we did some performance tests recently comparing HDFS > and Ozone. > Depending on the set up you have and the applications you are running, > there are variant levels of perf gaps that are identified, and I think > there are more critical features we need to implement than caching which > will improve performance more. > > If you run Impala, Impala supports remote caching which works on several > file systems including Ozone, and it helps with improving performance quite > nicely. > > On Wed, Jan 6, 2021 at 10:24 PM Lin, Yiqun <yiq...@ebay.com.invalid> > wrote: > > > Hi Michel, > > > > As far as I know, currently Ozone hasn't a feature like HDFS cache block. > > Not sure if someone else has done the performance test between Ozone read > > and HDFS read. > > But anyway, cache block on Ozone seems a good idea if it can greatly > speed > > the data read. > > > > Thanks, > > Yiqun > > > > On 2021/1/6, 11:47 PM, "Michel Sumbul" <michelsum...@gmail.com> wrote: > > > > External Email > > > > Hi guys, > > > > I would like to know if Ozone has a caching feature like HDFS? ( > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhadoop.apache.org%2Fdocs%2Fcurrent%2Fhadoop-project-dist%2Fhadoop-hdfs%2FCentralizedCacheManagement.html&data=04%7C01%7Cyiqlin%40ebay.com%7Ca80ab0cd29194003f00008d8b25a6411%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637455448585079421%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=MvuLw8eEANxVuHiCEprR%2BWkb%2FriKjhOoDlZPaX%2FEPVo%3D&reserved=0 > > ) > > > > I didn’t find any doc on it for Ozone, but caching blocks frequently > > access > > really improve the performance on HDFS and it might be a nice to have > > for > > Ozone to avoid unnecessary IOPS. > > > > Thanks, > > Michel > > > > PS: sorry for the resend, it was the old mailing list > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > > For additional commands, e-mail: dev-h...@ozone.apache.org > > >
