Thanks for sharing the GC link Michel! Memory Total heap (usage / alloc. max) 1,501.7M (81.7%) / 1,837.4M Tenured heap (usage / alloc. max) 1,361.7M (80.7%) / 1,686.7M Young heap (usage / alloc. max) 150.6M (100.0%) / 150.6M
I think the heap settings are not reflecting. Can you please try again? Please use -Xms and -Xmx both and set them to 8g. Also please try setting minimum young gen allocation to 2GB. Can you also please mention what tests are you comparing? Regards Lokesh > On 30-Jul-2020, at 8:33 PM, Michel Sumbul <michelsum...@gmail.com> wrote: > > Hi Lokesh, > > Thanks for your advice. > I changed the heap size from 3GB to 4Gb and then to 8GB, set to > dfs.ratis.client.request.retry to 10 and decreased the number of pipeline > from 5 to 2. > It allows 10-15% better performance but it's still 35% slower than HDFS. I > also notice that the data is less well spread over the 4 nodes with only 2 > pipelines. > > I had a look at the GC log during the execution of the teragen (copy attach > this email), I'm not an expert at all in gc tuning so I load it the GC log on > this site and it tell me that for the 4 minutes of the execution of the job > the datanode was in pause during 29 seconds and trigger 1260 GC over 4 > minutes. That looks pretty high for me but maybe not? > http://gcloganalyzer.com/?file=70338453-4505-4238-a900-c0206a2d52f4test.gc > > I also take care that the yarn job don't request more than 80% of the VM > resource (even try with 60% but didn't change anything) > > Do you think there is something else I can do to improve it or should I stop > here? Is it possible that the difference comes from the short circuit feature > of HDFS? > > Thanks a lot, > Michel > > Le mer. 29 juil. 2020 à 10:54, Lokesh Jain <lj...@apache.org> a écrit : > Hi Michel > > Thanks for trying out Ozone! > For ozone 0.5 - Can you please try another run after increasing config value > for dfs.ratis.client.request.retry.interval to 10 or 15 seconds? The default > value for this config is 1 second. > > ozone.datanode.pipeline.limit > Can you try a smaller value like 1 or 2 for above config with a data node > heap size of 4GB? Please check GC pressure on the datanode with this config. > > There are some improvements which have gone recently after ozone 0.5 release. > I would also recommend to try the latest ozone. > > Thanks > Lokesh > > > > On 29-Jul-2020, at 12:57 AM, Michel Sumbul <michelsum...@gmail.com> wrote: > > > > I forgot to mention that I set ozone.datanode.pipeline.limit to 5. > > Michel > > > > Le mar. 28 juil. 2020 à 20:22, Michel Sumbul <michelsum...@gmail.com > > <mailto:michelsum...@gmail.com>> a écrit : > > Hi guys, > > > > I would like to know if you have any advice tips/tricks to get the best > > performance of Ozone? (Memory tuning / thread / specific settings / etc..) > > > > I did a few teragen/terasort on it and the results are really surprising > > compared to HDFS,Ozone (using the hadoopFS) is almost 2 times slower than > > HDFS. > > > > > > > > > > > > > > The clusters were exactly the same for both: > > - 3 masters and 4 slaves (8core/32GB) (its a small cluster but that should > > matter here) > > - Backend storage is a CEPH storage (80 servers) > > - NIC: 2 x 25Gb/S > > - Ozone version 0.5 > > - Each job was executed 5 times > > > > HDFS and Ozone were installed on the same nodes, one was down where the > > other one was up, to guarantee no other differences of configuration that > > the distributed FS. > > > > I was not expecting a big difference like this, do you have any idea how to > > improve this? > > Or what can be the reason for that? I saw a few jira regarding data > > locality at read, can it be linked to that? > > > > Thanks, > > Michel > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
