Hi Lou
There is no direct memory copy, the transfer goes over TCP, even when processes
exevute on the same VM. So this means that the whole processing including
datatransfer is performed by the CPU.
So I would expect the high CPU load while transferring data.
Best
Adrian
Am 27. Februar 2020 15:32:43 MEZ schrieb Lou DeGenaro <lou.degen...@gmail.com>:
>Hi Adrian,
>
>client + namenode + datanode are all in same VM running in cloud. I
>think
>I'm using TCP, but I'm a new here, so not sure. Is there is another
>way
>that Crail does the data xfer memory-to-memory on same VM?
>
>The loop has nothing to do except wait for data xfer, so I would have
>expected CPU usage to be close to zero...no?
>
>execution time 1.048 (sec) to xfer 1GB, with CPU use > 100%
>
>crail-site.conf
>crail.namenode.address crail://sgt-pepper:9060
>crail.cachepath /tmp/hugepages/cache
>crail.cachelimit 1073741824
>crail.storage.tcp.interface eth0
>crail.storage.tcp.datapath /tmp/hugepages/data
>crail.storage.tcp.storagelimit 1073741824
>
>Lou.
>
>On Thu, Feb 27, 2020 at 9:15 AM Adrian Schuepbach <
>adrian.schuepb...@gribex.net> wrote:
>
>> Hi Lou
>>
>> I guess executing the loop consumes the cpu time.
>>
>> I assume your client + namenode + datanodes are connected by TCP.
>> So when you execute a tight loop I think it is normal to consume
>100%.
>>
>> Thanks
>> Adrian
>>
>>
>> On 2/27/20 15:08, Lou DeGenaro wrote:
>> > I have a simple loop in my Crail java client to fetch the contents
>of a
>> 1GB
>> > file like so:
>> >
>> > private CrailBuffer buf = null;
>> > int bufsize = 1024*1024;
>> > int filesize = 1024*1024*1024;
>> > ...
>> > buf = OffHeapBuffer.wrap(ByteBuffer.allocateDirect(bufsize));
>> > long sumbytes = 0;
>> > while (sumbytes < filesize) {
>> > buf.clear();
>> > CrailResult cr = directStream.read(buf).get();
>> > long ret = cr.getLen();
>> > sumbytes = sumbytes + ret;
>> > }
>> >
>> > The namenode, datanode, and my java client are all running in the
>same
>> > cloud VM. When running, 100% (and more) CPU is consumed by my java
>> app. I
>> > was expecting something close to zero. Why so much CPU? A
>consequence
>> of
>> > running in a VM or am I doing something wrong?
>> >
>> > Lou.
>> >
>> --
>> Adrian Schüpbach, Dr. sc. ETH Zürich
>>
>>
