Hi Adrian,
client + namenode + datanode are all in same VM running in cloud. I think
I'm using TCP, but I'm a new here, so not sure. Is there is another way
that Crail does the data xfer memory-to-memory on same VM?
The loop has nothing to do except wait for data xfer, so I would have
expected CPU usage to be close to zero...no?
execution time 1.048 (sec) to xfer 1GB, with CPU use > 100%
crail-site.conf
crail.namenode.address crail://sgt-pepper:9060
crail.cachepath /tmp/hugepages/cache
crail.cachelimit 1073741824
crail.storage.tcp.interface eth0
crail.storage.tcp.datapath /tmp/hugepages/data
crail.storage.tcp.storagelimit 1073741824
Lou.
On Thu, Feb 27, 2020 at 9:15 AM Adrian Schuepbach <
adrian.schuepb...@gribex.net> wrote:
> Hi Lou
>
> I guess executing the loop consumes the cpu time.
>
> I assume your client + namenode + datanodes are connected by TCP.
> So when you execute a tight loop I think it is normal to consume 100%.
>
> Thanks
> Adrian
>
>
> On 2/27/20 15:08, Lou DeGenaro wrote:
> > I have a simple loop in my Crail java client to fetch the contents of a
> 1GB
> > file like so:
> >
> > private CrailBuffer buf = null;
> > int bufsize = 1024*1024;
> > int filesize = 1024*1024*1024;
> > ...
> > buf = OffHeapBuffer.wrap(ByteBuffer.allocateDirect(bufsize));
> > long sumbytes = 0;
> > while (sumbytes < filesize) {
> > buf.clear();
> > CrailResult cr = directStream.read(buf).get();
> > long ret = cr.getLen();
> > sumbytes = sumbytes + ret;
> > }
> >
> > The namenode, datanode, and my java client are all running in the same
> > cloud VM. When running, 100% (and more) CPU is consumed by my java
> app. I
> > was expecting something close to zero. Why so much CPU? A consequence
> of
> > running in a VM or am I doing something wrong?
> >
> > Lou.
> >
> --
> Adrian Schüpbach, Dr. sc. ETH Zürich
>
>
