Lou,
Did you try the commands I gave you? That’s the simplest way of determining
what’s going on. You can also move files into the datastore using the fs
command
set. That way you can see better what’s going on. It appears from the error
that
you’ve exceeded the size of the memory (i.e. storage). I told you it was small
😊
You could try something smaller if you’d rather start with iobench perhaps.
Regards,
David
From: Lou DeGenaro<mailto:lou.degen...@gmail.com>
Sent: Friday, February 7, 2020 12:15 PM
To: dev@crail.apache.org<mailto:dev@crail.apache.org>
Subject: iobench
Still a noob. Got namenode and datanode running as docker images on my
VM. I'm looking for the simplest example of writing something then reading
something.
[root@abbey-road conf]# $CRAIL_HOME/bin/crail iobench -t write -f /filename
-s 1024 -k 1
20/02/07 14:10:22 INFO crail: creating singleton crail file system
20/02/07 14:10:23 INFO crail: crail.version 3101
20/02/07 14:10:23 INFO crail: crail.directorydepth 16
20/02/07 14:10:23 INFO crail: crail.tokenexpiration 10
20/02/07 14:10:23 INFO crail: crail.blocksize 1048576
20/02/07 14:10:23 INFO crail: crail.cachelimit 1073741824
20/02/07 14:10:23 INFO crail: crail.cachepath /dev/hugepages/cache
20/02/07 14:10:23 INFO crail: crail.user crail
20/02/07 14:10:23 INFO crail: crail.shadowreplication 1
20/02/07 14:10:23 INFO crail: crail.debug false
20/02/07 14:10:23 INFO crail: crail.statistics true
20/02/07 14:10:23 INFO crail: crail.rpctimeout 1000
20/02/07 14:10:23 INFO crail: crail.datatimeout 1000
20/02/07 14:10:23 INFO crail: crail.buffersize 1048576
20/02/07 14:10:23 INFO crail: crail.slicesize 524288
20/02/07 14:10:23 INFO crail: crail.singleton true
20/02/07 14:10:23 INFO crail: crail.regionsize 1073741824
20/02/07 14:10:23 INFO crail: crail.directoryrecord 512
20/02/07 14:10:23 INFO crail: crail.directoryrandomize true
20/02/07 14:10:23 INFO crail: crail.cacheimpl
org.apache.crail.memory.MappedBufferCache
20/02/07 14:10:23 INFO crail: crail.locationmap
20/02/07 14:10:23 INFO crail: crail.namenode.address crail://abbey-road:9060
20/02/07 14:10:23 INFO crail: crail.namenode.blockselection roundrobin
20/02/07 14:10:23 INFO crail: crail.namenode.fileblocks 16
20/02/07 14:10:23 INFO crail: crail.namenode.rpctype
org.apache.crail.namenode.rpc.tcp.TcpNameNode
20/02/07 14:10:23 INFO crail: crail.namenode.log
20/02/07 14:10:23 INFO crail: crail.storage.types
org.apache.crail.storage.tcp.TcpStorageTier
20/02/07 14:10:23 INFO crail: crail.storage.classes 1
20/02/07 14:10:23 INFO crail: crail.storage.rootclass 0
20/02/07 14:10:23 INFO crail: crail.storage.keepalive 2
20/02/07 14:10:23 INFO crail: buffer cache, allocationCount 1, bufferCount
1024
20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth 16,
messageSize 2097152, nodealy false
20/02/07 14:10:23 INFO crail: crail.storage.tcp.interface eth0
20/02/07 14:10:23 INFO crail: crail.storage.tcp.port 50020
20/02/07 14:10:23 INFO crail: crail.storage.tcp.storagelimit 1073741824
20/02/07 14:10:23 INFO crail: crail.storage.tcp.allocationsize 1073741824
20/02/07 14:10:23 INFO crail: crail.storage.tcp.datapath /dev/hugepages/data
20/02/07 14:10:23 INFO crail: crail.storage.tcp.queuedepth 16
20/02/07 14:10:23 INFO crail: crail.storage.tcp.cores 1
20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth 32,
messageSize 512, nodealy true
20/02/07 14:10:23 INFO crail: crail.namenode.tcp.queueDepth 32
20/02/07 14:10:23 INFO crail: crail.namenode.tcp.messageSize 512
20/02/07 14:10:23 INFO crail: crail.namenode.tcp.cores 1
20/02/07 14:10:23 INFO crail: connected to namenode(s) abbey-road/
10.114.222.23:9060
write, filename /filename, size 1024, loop 1, storageClass 0, locationClass
0, buffered true
Exception in thread "main" java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)
at
org.apache.crail.memory.MappedBufferCache.allocateRegion(MappedBufferCache.java:94)
at org.apache.crail.memory.BufferCache.allocateBuffer(BufferCache.java:95)
at
org.apache.crail.core.CoreDataStore.allocateBuffer(CoreDataStore.java:482)
at org.apache.crail.tools.CrailBenchmark.write(CrailBenchmark.java:87)
at org.apache.crail.tools.CrailBenchmark.main(CrailBenchmark.java:1070)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)
... 5 more
