Hi Lokesh, Thanks for your advice. I changed the heap size from 3GB to 4Gb and then to 8GB, set to dfs.ratis.client.request.retry to 10 and decreased the number of pipeline from 5 to 2. It allows 10-15% better performance but it's still 35% slower than HDFS. I also notice that the data is less well spread over the 4 nodes with only 2 pipelines.
I had a look at the GC log during the execution of the teragen (copy attach this email), I'm not an expert at all in gc tuning so I load it the GC log on this site and it tell me that for the 4 minutes of the execution of the job the datanode was in pause during 29 seconds and trigger 1260 GC over 4 minutes. That looks pretty high for me but maybe not? http://gcloganalyzer.com/?file=70338453-4505-4238-a900-c0206a2d52f4test.gc I also take care that the yarn job don't request more than 80% of the VM resource (even try with 60% but didn't change anything) Do you think there is something else I can do to improve it or should I stop here? Is it possible that the difference comes from the short circuit feature of HDFS? Thanks a lot, Michel Le mer. 29 juil. 2020 à 10:54, Lokesh Jain <lj...@apache.org> a écrit : > Hi Michel > > Thanks for trying out Ozone! > For ozone 0.5 - Can you please try another run after increasing config > value for dfs.ratis.client.request.retry.interval to 10 or 15 seconds? The > default value for this config is 1 second. > > ozone.datanode.pipeline.limit > Can you try a smaller value like 1 or 2 for above config with a data node > heap size of 4GB? Please check GC pressure on the datanode with this config. > > There are some improvements which have gone recently after ozone 0.5 > release. I would also recommend to try the latest ozone. > > Thanks > Lokesh > > > > On 29-Jul-2020, at 12:57 AM, Michel Sumbul <michelsum...@gmail.com> > wrote: > > > > I forgot to mention that I set ozone.datanode.pipeline.limit to 5. > > Michel > > > > Le mar. 28 juil. 2020 à 20:22, Michel Sumbul <michelsum...@gmail.com > <mailto:michelsum...@gmail.com>> a écrit : > > Hi guys, > > > > I would like to know if you have any advice tips/tricks to get the best > performance of Ozone? (Memory tuning / thread / specific settings / etc..) > > > > I did a few teragen/terasort on it and the results are really surprising > compared to HDFS,Ozone (using the hadoopFS) is almost 2 times slower than > HDFS. > > > > > > > > > > > > > > The clusters were exactly the same for both: > > - 3 masters and 4 slaves (8core/32GB) (its a small cluster but that > should matter here) > > - Backend storage is a CEPH storage (80 servers) > > - NIC: 2 x 25Gb/S > > - Ozone version 0.5 > > - Each job was executed 5 times > > > > HDFS and Ozone were installed on the same nodes, one was down where the > other one was up, to guarantee no other differences of configuration that > the distributed FS. > > > > I was not expecting a big difference like this, do you have any idea how > to improve this? > > Or what can be the reason for that? I saw a few jira regarding data > locality at read, can it be linked to that? > > > > Thanks, > > Michel > > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: ozone-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-dev-h...@hadoop.apache.org
