As a rule of thumb for sizing purposes, we should have 1000 MB memory for one million blocks.
Thanks, Ramdas On Tue, Jan 29, 2019 at 5:53 PM Wei-Chiu Chuang <weic...@cloudera.com.invalid> wrote: > I don't feel this is strictly a small file issue (since I am not seeing > the average file size) > But it looks like your directory/file ratio is way too low. I've seen that > when Hive creates too many partitions. That can render Hive queries > inefficient. > > On Tue, Jan 29, 2019 at 2:09 PM Sudhir Babu Pothineni < > sbpothin...@gmail.com> wrote: > >> >> One of Hadoop cluster I am working >> >> 85,985,789 files and directories, 58,399,919 blocks = 144,385,717 total >> file system objects >> >> Heap memory used 132.0 GB of 256 GB Heap Memory. >> >> I feel it’s odd the ratio of files vs blocks way higher showing more of >> small files problem, >> >> But the cluster working fine. Am I worrying unnecessarily? we are using >> Hadoop 2.6.0 >> >> Thanks >> Sudhir >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: user-h...@hadoop.apache.org >> >>
