Re: Question on crail type-1 with terasort shuffle data

Wed, 26 Jun 2019 09:18:07 -0700

There can only be 1 NQN in the whole system. For multiple storage classes just use different ip/port combinations with the same NQN. e.g. start a NVMf server on node 1 with storage class 1 with NQN X and start a server on node 2 with storage class 2 with NQN X. You can start another server on node 1 by choosing a different port but still use the same NQN X. I know that this is cumbersome to use. We are working on it, we just have not found a solution that makes everyone happy :-) 

Regards,
Jonas

 On Wed, 26 Jun 2019 16:09:06 +0000
 David Crespi <david.cre...@storedgesystems.com> wrote:

Sorry Jonas but I’m still confused.

I started off doing this by having each storage class 1 connected to 
it’s own subsystem,
with a single namespace.  This is where I was seeing the correct IP 
address of that subsystem,
but the wrong controller name.  It was the controller name of the 
storage class 2, that was also
registered with the namenode.  So this is the correct way of using 
it, but it’s having a problem

determining which controller to identify?

Regards,

          David

From: Jonas Pfefferle<mailto:peppe...@japf.ch>
Sent: Wednesday, June 26, 2019 9:01 AM

To: dev@crail.apache.org<mailto:dev@crail.apache.org>; David 
Crespi<mailto:david.cre...@storedgesystems.com>

Subject: Re: Question on crail type-1 with terasort shuffle data


I'm sure you already figured this out but just to make sure we are 
on the
same page. The NVMf storage tier (or datanode, we use them 
interchangeably)
is just another NVMf client that connects to the NVMf server and 
retrieves
all necessary information like capacity etc. to register the 
datanode to the
namenode. When starting a NVMf datanode you can specify via the 
commandline
what storage class the datanode should be in. It then registers the 
ip/port
combination with the storage class information at the namenode. Now 
whenever
a block is requested from the namenode by a client, the namenode 
sends the

ip/port information and the storage class to the client. With this

information the client connects to the NVMf server (not the 
datanode)
directly with the NQN provided in the config file, namespace 1 and 
the

ip/port information it got from the namenode.


I hope this makes it clearer.

Regards,
Jonas

 On Wed, 26 Jun 2019 14:55:05 +0000
 David Crespi <david.cre...@storedgesystems.com> wrote:

Yep.  I wanted to use crail-store for temp results storage, storage
class 2, AND the shuffle data, with

storage class 1.


Regards,


          David


________________________________
From: Jonas Pfefferle <peppe...@japf.ch>
Sent: Wednesday, June 26, 2019 7:51:43 AM
To: dev@crail.apache.org; David Crespi
Subject: Re: Question on crail type-1 with terasort shuffle data

David,


Let's make sure we talk about the same thing, I assume with type 1
and 2 you
mean storage classes?

Regards,
Jonas

 On Wed, 26 Jun 2019 14:25:52 +0000
 David Crespi <david.cre...@storedgesystems.com> wrote:

I’m a bit confused now. If it only ever handles 1 namespace, then
multiple datanodes are useless.

Or are you saying only 1 namespace for type-1?  If you’re not saying
only one for type-1, how do you

mix types if limited to 1 namespace?


Regards,


          David


________________________________
From: Jonas Pfefferle <peppe...@japf.ch>
Sent: Wednesday, June 26, 2019 12:25:38 AM
To: dev@crail.apache.org; David Crespi
Subject: Re: Question on crail type-1 with terasort shuffle data

I forgot to add: Namespace has to be 1 for the moment. This is a
limitation
in how the namenode handles storage tier metadata (essentially it
only
stores ip/port). You can set it on the server side however it will
not be
respected by the client, i.e. the client will always use namespace
id 1.


Regards,
Jonas

 On Wed, 26 Jun 2019 09:17:40 +0200
 "Jonas Pfefferle" <peppe...@japf.ch> wrote:

Hi David,


This is currently a limitation of the NVMf storage tier. You need to
use the same NQN on all datanodes. The port can be different however.
I will clarify this in the documentation.

Regards,
Jonas

 On Tue, 25 Jun 2019 19:43:04 +0000
 David Crespi <david.cre...@storedgesystems.com> wrote:

It occurred to me that perhaps since I’m using the NVMf, that the
namenode can only handle one

Subsystem.  And for the type-1, these resources would need to be
added under the same subsystem,

but as different namespace’s.  Is this correct?  I did try this and
it appears to be working. If this IS the

correct way of utilizing these, how do I know which namespace(s) are
being accessed?  I don’t see

anything in the logs that identify which is being addressed.


Regards,


          David


________________________________
From: David Crespi <david.cre...@storedgesystems.com>
Sent: Tuesday, June 25, 2019 11:09:56 AM
To: dev@crail.apache.org
Subject: Question on crail type-1 with terasort shuffle data

I’m starting to run terasort now with the shuffle data going to
crail type-1.
I’ve got the type-2 set to nqn:
nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
which appears to be working okay for the result data.  When I
introduce type-1 into my config,
it looks like the namenode gets confused and picks the nqn of the
type-2 crailstore, instead of
the one it is assigned (bolded in red below).

In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
then 5 type-1 added to the
spark worker nodes (192.168.3.101-105).  All are running the same
image as the namenode,
which is the spark master.

Is there some additional setting I need to include for this?
These are the variables that get set for each container (added via
environment)
# NVMf storage
crail.storage.nvmf.ip                     $NVMF_IP
crail.storage.nvmf.port                 $NVMF_PORT
crail.storage.nvmf.nqn                  $NVMF_NQN
crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
crail.storage.nvmf.namespace      $NVMF_NAMESPACE

19/06/25 10:46:43 INFO crail: creating singleton crail file system
19/06/25 10:46:43 INFO crail: crail.version 3101
19/06/25 10:46:43 INFO crail: crail.directorydepth 16
19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
19/06/25 10:46:43 INFO crail: crail.cachelimit 0
19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
19/06/25 10:46:43 INFO crail: crail.user crail
19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
19/06/25 10:46:43 INFO crail: crail.debug true
19/06/25 10:46:43 INFO crail: crail.statistics true
19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
19/06/25 10:46:43 INFO crail: crail.slicesize 65536
19/06/25 10:46:43 INFO crail: crail.singleton true
19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
19/06/25 10:46:43 INFO crail: crail.cacheimpl
org.apache.crail.memory.MappedBufferCache
19/06/25 10:46:43 INFO crail: crail.locationmap
19/06/25 10:46:43 INFO crail: crail.namenode.address
crail://192.168.1.164:9060
19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
roundrobin
19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
org.apache.crail.namenode.rpc.tcp.TcpNameNode
19/06/25 10:46:43 INFO crail: crail.namenode.log
19/06/25 10:46:43 INFO crail: crail.storage.types
org.apache.crail.storage.nvmf.NvmfStorageTier
19/06/25 10:46:43 INFO crail: crail.storage.classes 2
19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
bufferCount 1024
19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
1073741824
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
queueDepth 32, messageSize 512, nodealy true
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
19/06/25 10:46:43 INFO crail: connected to namenode(s)
/192.168.1.164:9060
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
streamId 1, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
address = /192.168.3.100:4420, subsystem NQN =
nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
19/06/25 10:46:43 INFO disni: jverbs jni version 32
19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
size 28, native size 16
19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
native size 32
19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
72, native size 128
19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
native size 48
19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
native size 16
19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
40, native size 40
19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
native size 48
19/06/25 10:46:43 INFO disni: createEventChannel, objId
139964194703408
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: createId, id 139964194793600
19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres
/192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
19/06/25 10:46:43 INFO disni: setting up protection domain, context
461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context
139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
64
19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type +
RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
/192.168.3.100:4420
19/06/25 10:46:43 INFO disni: createId, id 139964195036000
19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres
/192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context
461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context
139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
128
19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
send_wr size 64, recv_wr_size 64
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type +
RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
/192.168.3.100:4420
19/06/25 10:46:43 INFO crail: EndpointCache miss
/192.168.3.100:4420, fsId 0, cache size 1
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
success
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
streamId 1, capacity 262656
19/06/25 10:46:43 INFO crail: createNode: name /spark, type
DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
streamId 2, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
fsId 0
19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
4, token 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
streamId 2, capacity 524800
19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
type DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
fd 4, streamId 3, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
address = /192.168.3.104:4420, subsystem NQN =
nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
nqn.2018-12.com.StorEdgeSystems:worker-4)
19/06/25 10:46:43 INFO disni: createEventChannel, objId
139964195131168
19/06/25 10:46:43 INFO disni: createId, id 139964195141520
19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres
/192.168.3.104:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context
461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context
139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
64
19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type +
RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
/192.168.3.104:4420
19/06/25 10:46:43 INFO crail: ERROR: failed data operation
com.ibm.jnvmf.UnsuccessfulComandException: Command was not
successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
The host is not allowed to establish an association to any controller
in the NVM subsystem or the host is not allowed to establish an
association to the specified controller., CID: 0, Do_not_retry:
false, More: false, SQHD: 0}
       at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
       at
com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)


Regards,

          David



























					Reply via email to