We’re talking host nqn right, not subsystem nqn? David 714-476-2692
> On Jun 26, 2019, at 09:17, Jonas Pfefferle <peppe...@japf.ch> wrote: > > There can only be 1 NQN in the whole system. For multiple storage classes > just use different ip/port combinations with the same NQN. e.g. start a NVMf > server on node 1 with storage class 1 with NQN X and start a server on node 2 > with storage class 2 with NQN X. You can start another server on node 1 by > choosing a different port but still use the same NQN X. > I know that this is cumbersome to use. We are working on it, we just have not > found a solution that makes everyone happy :-) > > > Regards, > Jonas > > On Wed, 26 Jun 2019 16:09:06 +0000 > David Crespi <david.cre...@storedgesystems.com> wrote: >> Sorry Jonas but I’m still confused. >> I started off doing this by having each storage class 1 connected to it’s >> own subsystem, >> with a single namespace. This is where I was seeing the correct IP address >> of that subsystem, >> but the wrong controller name. It was the controller name of the storage >> class 2, that was also >> registered with the namenode. So this is the correct way of using it, but >> it’s having a problem >> determining which controller to identify? >> Regards, >> David >> From: Jonas Pfefferle<mailto:peppe...@japf.ch> >> Sent: Wednesday, June 26, 2019 9:01 AM >> To: dev@crail.apache.org<mailto:dev@crail.apache.org>; David >> Crespi<mailto:david.cre...@storedgesystems.com> >> Subject: Re: Question on crail type-1 with terasort shuffle data >> I'm sure you already figured this out but just to make sure we are on the >> same page. The NVMf storage tier (or datanode, we use them interchangeably) >> is just another NVMf client that connects to the NVMf server and retrieves >> all necessary information like capacity etc. to register the datanode to the >> namenode. When starting a NVMf datanode you can specify via the commandline >> what storage class the datanode should be in. It then registers the ip/port >> combination with the storage class information at the namenode. Now whenever >> a block is requested from the namenode by a client, the namenode sends the >> ip/port information and the storage class to the client. With this >> information the client connects to the NVMf server (not the datanode) >> directly with the NQN provided in the config file, namespace 1 and the >> ip/port information it got from the namenode. >> I hope this makes it clearer. >> Regards, >> Jonas >> On Wed, 26 Jun 2019 14:55:05 +0000 >> David Crespi <david.cre...@storedgesystems.com> wrote: >>> Yep. I wanted to use crail-store for temp results storage, storage >>> class 2, AND the shuffle data, with >>> >>> storage class 1. >>> >>> >>> Regards, >>> >>> >>> David >>> >>> >>> ________________________________ >>> From: Jonas Pfefferle <peppe...@japf.ch> >>> Sent: Wednesday, June 26, 2019 7:51:43 AM >>> To: dev@crail.apache.org; David Crespi >>> Subject: Re: Question on crail type-1 with terasort shuffle data >>> >>> David, >>> >>> >>> Let's make sure we talk about the same thing, I assume with type 1 >>> and 2 you >>> mean storage classes? >>> >>> Regards, >>> Jonas >>> >>> On Wed, 26 Jun 2019 14:25:52 +0000 >>> David Crespi <david.cre...@storedgesystems.com> wrote: >>>> I’m a bit confused now. If it only ever handles 1 namespace, then >>>> multiple datanodes are useless. >>>> >>>> Or are you saying only 1 namespace for type-1? If you’re not saying >>>> only one for type-1, how do you >>>> >>>> mix types if limited to 1 namespace? >>>> >>>> >>>> Regards, >>>> >>>> >>>> David >>>> >>>> >>>> ________________________________ >>>> From: Jonas Pfefferle <peppe...@japf.ch> >>>> Sent: Wednesday, June 26, 2019 12:25:38 AM >>>> To: dev@crail.apache.org; David Crespi >>>> Subject: Re: Question on crail type-1 with terasort shuffle data >>>> >>>> I forgot to add: Namespace has to be 1 for the moment. This is a >>>> limitation >>>> in how the namenode handles storage tier metadata (essentially it >>>> only >>>> stores ip/port). You can set it on the server side however it will >>>> not be >>>> respected by the client, i.e. the client will always use namespace >>>> id 1. >>>> >>>> >>>> Regards, >>>> Jonas >>>> >>>> On Wed, 26 Jun 2019 09:17:40 +0200 >>>> "Jonas Pfefferle" <peppe...@japf.ch> wrote: >>>>> Hi David, >>>>> >>>>> >>>>> This is currently a limitation of the NVMf storage tier. You need to >>>>> use the same NQN on all datanodes. The port can be different however. >>>>> I will clarify this in the documentation. >>>>> >>>>> Regards, >>>>> Jonas >>>>> >>>>> On Tue, 25 Jun 2019 19:43:04 +0000 >>>>> David Crespi <david.cre...@storedgesystems.com> wrote: >>>>>> It occurred to me that perhaps since I’m using the NVMf, that the >>>>>> namenode can only handle one >>>>>> >>>>>> Subsystem. And for the type-1, these resources would need to be >>>>>> added under the same subsystem, >>>>>> >>>>>> but as different namespace’s. Is this correct? I did try this and >>>>>> it appears to be working. If this IS the >>>>>> >>>>>> correct way of utilizing these, how do I know which namespace(s) are >>>>>> being accessed? I don’t see >>>>>> >>>>>> anything in the logs that identify which is being addressed. >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> >>>>>> David >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: David Crespi <david.cre...@storedgesystems.com> >>>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM >>>>>> To: dev@crail.apache.org >>>>>> Subject: Question on crail type-1 with terasort shuffle data >>>>>> >>>>>> I’m starting to run terasort now with the shuffle data going to >>>>>> crail type-1. >>>>>> I’ve got the type-2 set to nqn: >>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100, >>>>>> which appears to be working okay for the result data. When I >>>>>> introduce type-1 into my config, >>>>>> it looks like the namenode gets confused and picks the nqn of the >>>>>> type-2 crailstore, instead of >>>>>> the one it is assigned (bolded in red below). >>>>>> >>>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and >>>>>> then 5 type-1 added to the >>>>>> spark worker nodes (192.168.3.101-105). All are running the same >>>>>> image as the namenode, >>>>>> which is the spark master. >>>>>> >>>>>> Is there some additional setting I need to include for this? >>>>>> These are the variables that get set for each container (added via >>>>>> environment) >>>>>> # NVMf storage >>>>>> crail.storage.nvmf.ip $NVMF_IP >>>>>> crail.storage.nvmf.port $NVMF_PORT >>>>>> crail.storage.nvmf.nqn $NVMF_NQN >>>>>> crail.storage.nvmf.hostnqn $NVMF_HOSTNQN >>>>>> crail.storage.nvmf.namespace $NVMF_NAMESPACE >>>>>> >>>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system >>>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101 >>>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16 >>>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10 >>>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576 >>>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0 >>>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache >>>>>> 19/06/25 10:46:43 INFO crail: crail.user crail >>>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1 >>>>>> 19/06/25 10:46:43 INFO crail: crail.debug true >>>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true >>>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000 >>>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000 >>>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576 >>>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536 >>>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true >>>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824 >>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512 >>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true >>>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl >>>>>> org.apache.crail.memory.MappedBufferCache >>>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address >>>>>> crail://192.168.1.164:9060 >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection >>>>>> roundrobin >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16 >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype >>>>>> org.apache.crail.namenode.rpc.tcp.TcpNameNode >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types >>>>>> org.apache.crail.storage.nvmf.NvmfStorageTier >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2 >>>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0, >>>>>> bufferCount 1024 >>>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn >>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn >>>>>> nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize >>>>>> 1073741824 >>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64 >>>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0, >>>>>> queueDepth 32, messageSize 512, nodealy true >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32 >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512 >>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1 >>>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s) >>>>>> /192.168.1.164:9060 >>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark >>>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark >>>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2 >>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark >>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true >>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, >>>>>> streamId 1, isDir true, writeHint 0 >>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport >>>>>> address = /192.168.3.100:4420, subsystem NQN = >>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 >>>>>> 19/06/25 10:46:43 INFO disni: creating RdmaProvider of type 'nat' >>>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32 >>>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs >>>>>> size 28, native size 16 >>>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32, >>>>>> native size 32 >>>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size >>>>>> 72, native size 128 >>>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48, >>>>>> native size 48 >>>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16, >>>>>> native size 16 >>>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size >>>>>> 40, native size 40 >>>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48, >>>>>> native size 48 >>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId >>>>>> 139964194703408 >>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0 >>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600 >>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0 >>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres >>>>>> /192.168.3.100:4420 >>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080 >>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context >>>>>> 461, pd 1 >>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor >>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context >>>>>> 139962246807056 >>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe >>>>>> 64 >>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176, >>>>>> send_wr size 32, recv_wr_size 32 >>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: got event type + >>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress >>>>>> /192.168.3.100:4420 >>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000 >>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0 >>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres >>>>>> /192.168.3.100:4420 >>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context >>>>>> 461, pd 1 >>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor >>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context >>>>>> 139962246807056 >>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe >>>>>> 128 >>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304, >>>>>> send_wr size 64, recv_wr_size 64 >>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: got event type + >>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress >>>>>> /192.168.3.100:4420 >>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss >>>>>> /192.168.3.100:4420, fsId 0, cache size 1 >>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true, >>>>>> success >>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, >>>>>> streamId 1, capacity 262656 >>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type >>>>>> DIRECTORY, storageAffinity 0, locationAffinity 0 >>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, >>>>>> streamId 2, isDir true, writeHint 0 >>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420, >>>>>> fsId 0 >>>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd >>>>>> 4, token 0 >>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, >>>>>> streamId 2, capacity 524800 >>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast, >>>>>> type DIRECTORY, storageAffinity 0, locationAffinity 0 >>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark, >>>>>> fd 4, streamId 3, isDir true, writeHint 0 >>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport >>>>>> address = /192.168.3.104:4420, subsystem NQN = >>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 (<-- should be >>>>>> nqn.2018-12.com.StorEdgeSystems:worker-4) >>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId >>>>>> 139964195131168 >>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520 >>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0 >>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0 >>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres >>>>>> /192.168.3.104:4420 >>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context >>>>>> 461, pd 1 >>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor >>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context >>>>>> 139962246807056 >>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe >>>>>> 64 >>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896, >>>>>> send_wr size 32, recv_wr_size 32 >>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0 >>>>>> 19/06/25 10:46:43 INFO disni: got event type + >>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress >>>>>> /192.168.3.104:4420 >>>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation >>>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not >>>>>> successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 - >>>>>> The host is not allowed to establish an association to any controller >>>>>> in the NVM subsystem or the host is not allowed to establish an >>>>>> association to the specified controller., CID: 0, Do_not_retry: >>>>>> false, More: false, SQHD: 0} >>>>>> at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128) >>>>>> at >>>>>> com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36) >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> David >>>>>> >>>>> >
