Hi Daegyu, let's move this discussion to the user group, so that any one else can comment on this. I obviously don't have the best answers to the questions. But these are great questions.
Re: benchmarks for SCR: I believe yes. In fact, I found a benchmark running Accumulo and HBase on HDFS http://accumulosummit.com/2015/program/talks/hdfs-short-circuit-local-read-performance-benchmarking-with-apache-accumulo-and-apache-hbase/ However, HDFS SCR is a very old feature, and since this isn't new, there is less interest in performing the same benchmarks. So I don't expect to see new benchmarks. As far as I know I've not received reports regarding SCR performance regression for users running HBase or Impala (these two applications typically have SCR enabled). A colleague of mine, Nicolae (CC'ed here) is also doing a similar benchmark with NVMe SSD. I believe Nicolae will be interested in how you pushed HDSF to the hardware limit. IIRC the theoretical limit of DataNode to client is about 500MB/s per client. On Fri, Aug 30, 2019 at 11:49 PM Daegyu Han <hdg9...@gmail.com> wrote: > Sorry for late reply. > > I used my own benchmark using HDFS api. > > The cluster environment I used is as follows: > > Hadoop 2.9.2 > > Samsung nvme ssd > > Configured hdfs block size to 1GB. > > I used only one datanode to prevent remote read. > > > First, I uploaded 1Gb file to hdfs. > > Then, I ran my benchmark code. > > I added some log to my hdfs code so I can see each method runtime. > > > Anyway, Has there been any performance evaluation by companies using HDFS > on SCR and legacy read? > > > As far as I know, legacy read goes through datanode so it induce many > sendfile system call and tcp socket opening overhead. > > > Intuitively, I think SCR that client directly read file should be faster > than legacy read. > > However, the first step which is requesting file is synchronous and can be > overhead when using fast nvme ssd. > > > What do you think? > > > Thank you > > > > 2019년 8월 30일 (금) 22:27, Wei-Chiu Chuang <weic...@cloudera.com>님이 작성: > >> Interesting benchmark. Thank you, Daegyu. >> Can you try a larger file too? Like 128mb or 1gb? HDFS is not optimized >> for smaller files. >> >> What did you use for benchmark? >> >> Daegyu Han <hdg9...@gmail.com>於 2019年8月29日 週四，下午11:40寫道： >> >>> Hi all, >>> >>> Is ShortCircuit read faster than legacy read which goes through data >>> nodes? >>> >>> I have evaluated SCR and legacy local read on both HDD and NVMe SSD. >>> However, I have not seen any results that SCR is faster than legacy. >>> >>> Rather, SCR was slower than legacy when using NVMe SSDs because of the >>> operation that initially to get the file descriptors. >>> >>> When I configured SCR, getBlockReader() elapsed time is slower than >>> legacy local read. >>> >>> When I used NVMe SSD, >>> I also found that DFSInputStream: dataIn.read() time is really similar >>> to hardware limit. >>> (8MB/0.00289sec) = 2800MB/s >>> >>> I checked the logs that the execution time measured by the application >>> took 5ms to process 8mb. >>> >>> There is a 3 ms runtime difference between blockReader.doRead () in >>> DFSInputStream.java and dataIn.read () in BlockReader.java. >>> Where is this 3ms difference from? >>> >>> Thank you >>> ᐧ >>> >>
