Looks like both cmds are working, but it’s not really finding the datastore.
You’re getting info into the namenode, but not the datanode (which means
you’re really not writing to the datanode). The
First warning you can ignore (WARN NativeCodeLoader: Unable to load
native-hadoop library for your platform)
you may want to turn on debug to see more of what’s going on.
Edit /crail/conf/log4j.properties and change from INFO to DEBUG.
You should probably also look at the individual docker logs.
docker logs YourNamenodeName & docker logs YourDatanodeName
may give you some more hints of what’s happening.
Regards,
David
From: Lou DeGenaro<mailto:lou.degen...@gmail.com>
Sent: Monday, February 10, 2020 8:12 AM
To: dev@crail.apache.org<mailto:dev@crail.apache.org>
Subject: Re: iobench
OK, I take it back. Having trouble with crail fs.
=====
[root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -mkdir /foobar
20/02/10 10:07:16 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
...
20/02/10 10:07:16 INFO crail: connected to namenode(s) abbey-road/
10.114.222.23:9060
20/02/10 10:07:16 INFO crail: CrailHadoopFileSystem fs initialization done..
mkdir: java.util.concurrent.ExecutionException: java.io.IOException: Map
failed
20/02/10 10:07:16 INFO crail: Closing CrailHadoopFileSystem
20/02/10 10:07:16 INFO crail: Closing CrailFS singleton
20/02/10 10:07:16 INFO crail: mapped client cache closed
[root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -rmdir /foobar
20/02/10 10:08:55 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
...20/02/10 10:08:55 INFO crail: CrailHadoopFileSystem fs initialization
done..
rmdir: /foobar
20/02/10 10:08:55 INFO crail: Closing CrailHadoopFileSystem
20/02/10 10:08:55 INFO crail: Closing CrailFS singleton
20/02/10 10:08:55 INFO crail: mapped client cache closed
[root@abbey-road ~]# $CRAIL_HOME/bin/crail fs -mkdir /foobar
20/02/10 10:09:43 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
...
20/02/10 10:09:43 INFO crail: CrailHadoopFileSystem fs initialization done..
mkdir: `/foobar': File exists
20/02/10 10:09:43 INFO crail: Closing CrailHadoopFileSystem
20/02/10 10:09:43 INFO crail: Closing CrailFS singleton
20/02/10 10:09:43 INFO crail: mapped client cache closed
=====
Seems that mkdir works, but rmdir does not? Also, IOException was reported
on mkdir, which seems worrisome.
Lou.
On Mon, Feb 10, 2020 at 8:16 AM Lou DeGenaro <lou.degen...@gmail.com> wrote:
> David,
>
> Thanks. I'm able to use *crail fs *successfully to create/list/delete.
> Will try to increase available storage space next...
>
> Lou.
>
> On Fri, Feb 7, 2020 at 6:37 PM David Crespi <
> david.cre...@storedgesystems.com> wrote:
>
>> Oh, and it may have already written something on the datastore, so you’d
>> need
>> to check it and remove it with the fs command anyway.
>>
>> crail fs -ls -R /
>>
>> would show you what you have there.
>>
>> Regards,
>> David
>>
>> (C) 714-476-2692
>>
>> From: Lou DeGenaro<mailto:lou.degen...@gmail.com>
>> Sent: Friday, February 7, 2020 12:15 PM
>> To: dev@crail.apache.org<mailto:dev@crail.apache.org>
>> Subject: iobench
>>
>> Still a noob. Got namenode and datanode running as docker images on my
>> VM. I'm looking for the simplest example of writing something then
>> reading
>> something.
>>
>> [root@abbey-road conf]# $CRAIL_HOME/bin/crail iobench -t write -f
>> /filename
>> -s 1024 -k 1
>> 20/02/07 14:10:22 INFO crail: creating singleton crail file system
>> 20/02/07 14:10:23 INFO crail: crail.version 3101
>> 20/02/07 14:10:23 INFO crail: crail.directorydepth 16
>> 20/02/07 14:10:23 INFO crail: crail.tokenexpiration 10
>> 20/02/07 14:10:23 INFO crail: crail.blocksize 1048576
>> 20/02/07 14:10:23 INFO crail: crail.cachelimit 1073741824
>> 20/02/07 14:10:23 INFO crail: crail.cachepath /dev/hugepages/cache
>> 20/02/07 14:10:23 INFO crail: crail.user crail
>> 20/02/07 14:10:23 INFO crail: crail.shadowreplication 1
>> 20/02/07 14:10:23 INFO crail: crail.debug false
>> 20/02/07 14:10:23 INFO crail: crail.statistics true
>> 20/02/07 14:10:23 INFO crail: crail.rpctimeout 1000
>> 20/02/07 14:10:23 INFO crail: crail.datatimeout 1000
>> 20/02/07 14:10:23 INFO crail: crail.buffersize 1048576
>> 20/02/07 14:10:23 INFO crail: crail.slicesize 524288
>> 20/02/07 14:10:23 INFO crail: crail.singleton true
>> 20/02/07 14:10:23 INFO crail: crail.regionsize 1073741824
>> 20/02/07 14:10:23 INFO crail: crail.directoryrecord 512
>> 20/02/07 14:10:23 INFO crail: crail.directoryrandomize true
>> 20/02/07 14:10:23 INFO crail: crail.cacheimpl
>> org.apache.crail.memory.MappedBufferCache
>> 20/02/07 14:10:23 INFO crail: crail.locationmap
>> 20/02/07 14:10:23 INFO crail: crail.namenode.address
>> crail://abbey-road:9060
>> 20/02/07 14:10:23 INFO crail: crail.namenode.blockselection roundrobin
>> 20/02/07 14:10:23 INFO crail: crail.namenode.fileblocks 16
>> 20/02/07 14:10:23 INFO crail: crail.namenode.rpctype
>> org.apache.crail.namenode.rpc.tcp.TcpNameNode
>> 20/02/07 14:10:23 INFO crail: crail.namenode.log
>> 20/02/07 14:10:23 INFO crail: crail.storage.types
>> org.apache.crail.storage.tcp.TcpStorageTier
>> 20/02/07 14:10:23 INFO crail: crail.storage.classes 1
>> 20/02/07 14:10:23 INFO crail: crail.storage.rootclass 0
>> 20/02/07 14:10:23 INFO crail: crail.storage.keepalive 2
>> 20/02/07 14:10:23 INFO crail: buffer cache, allocationCount 1, bufferCount
>> 1024
>> 20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth
>> 16,
>> messageSize 2097152, nodealy false
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.interface eth0
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.port 50020
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.storagelimit 1073741824
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.allocationsize 1073741824
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.datapath
>> /dev/hugepages/data
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.queuedepth 16
>> 20/02/07 14:10:23 INFO crail: crail.storage.tcp.cores 1
>> 20/02/07 14:10:23 INFO narpc: new NaRPC client group v1.5.0, queueDepth
>> 32,
>> messageSize 512, nodealy true
>> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.queueDepth 32
>> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.messageSize 512
>> 20/02/07 14:10:23 INFO crail: crail.namenode.tcp.cores 1
>> 20/02/07 14:10:23 INFO crail: connected to namenode(s) abbey-road/
>> 10.114.222.23:9060
>> write, filename /filename, size 1024, loop 1, storageClass 0,
>> locationClass
>> 0, buffered true
>> Exception in thread "main" java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)
>> at
>>
>> org.apache.crail.memory.MappedBufferCache.allocateRegion(MappedBufferCache.java:94)
>> at org.apache.crail.memory.BufferCache.allocateBuffer(BufferCache.java:95)
>> at
>> org.apache.crail.core.CoreDataStore.allocateBuffer(CoreDataStore.java:482)
>> at org.apache.crail.tools.CrailBenchmark.write(CrailBenchmark.java:87)
>> at org.apache.crail.tools.CrailBenchmark.main(CrailBenchmark.java:1070)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)
>> ... 5 more
>>
>>
