[
https://issues.apache.org/jira/browse/CRAIL-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785875#comment-16785875
]
David Crespi commented on CRAIL-93:
-----------------------------------
I think we may be talking (writing) past each other.
If I don’t define the NVMF_NQN, crail defaults to nqn.2017-06.io.crail:cnode,
and expects a subsystem with this NQN. How I started this off is that we
generate our own subsystem NQN’s for the SPDK target subsystems. I wanted to
define this so Crail can find it… so I added in the variable
NVMF_NQN=”mySubSystemName”. Crail was appending the port, as you said, to that
name. If I don’t define that variable, then Crail expects to find a subsystem
with the default variable name. That’s all good. My point was, and I think
you’ve now stated this as well, is with the hack, it was requiring two
subsystems to exist for Crail to properly attach. mySubSystemName &
mySubSystemNamePORT#. If both does not exist with associated Namespaces, then
Crail will not attach. I’ve validated both scenarios and the behavior is as I
described. So I think we can put this issue to bed? And I look forward to the
patch.
Okay, now for host NQN/host ID’s. The clients will use host NQN’s to create a
single queue pair (qp) for the system to use in communicating with the target.
If, for example, a client is virtualized with different customer applications,
they may want to create unique environments (multi-tenant). The administrator
can assign a unique host ID to each of those tenants, along with the host NQN
and then each of those tenants with get a dedicated qp. What I had noticed,
and what the 2nd issue I stated in the original post was about the behavior of
Crail. Typically, if the target is configured with the option to “allow all
hosts”, then if the client attempts to discover and attach to that subsystem,
and has it’s host NQN/host ID as part of the command, then the target is happy
to allow the connection. The client, BTW is not, at this point, required to
include the host NQN/host ID to the connect command, since the host is allowing
all to connect… but if it’s there, the target doesn’t care. What I noticed,
however with Crail, is this the option set to “allow all hosts” on the
subsystems, AND Crail using it’s host NQN in the connect command, it didn’t
want to establish a connection… which is why I went to changing the subsystem
names to nqn.2017-06.io.crail:cnode and nqn.2017-06.io.crail:cnode4420. It was
pretty late when I was doing all this… so it may have been also due to the
issue with needed two subsystems in order to connect, but my memory says that I
had already discovered the dual subsystem “requirement” at that point. I will
also go back and validate this to be 100% sure! And BTW, I have validated that
the target will allow connections with a client that is adding it’s host NQN in
the connect command when the “allow all hosts” is set… so that’s also put to
bed!
To your logging point… completely understood!
To answer your container question… I had built my own containers with crail
months ago. Now that I’ve been playing with it more, I decided to use the
container you provided as the base for ours, then modified it as I needed to.
I figure in the long run, you guys know your software requirements way better
than I ever could eh?
My crail-site.conf file contains these entries for NVMf:
# NVMf storage
crail.storage.nvmf.ip $NVMF_IP
crail.storage.nvmf.port $NVMF_PORT
crail.storage.nvmf.nqn $NVMF_NQN
crail.storage.nvmf.namespace $NVMF_NAMESPACE
and then I start the container with these environment variables:
-e NVMF_IP="192.168.2.104"
-e NVMF_PORT="4420"
-e NVMF_NQN="nqn.2018-12.com.StorEdgeSystems:cntlr9"
-e NVMF_NAMESPACE="1"
As far at the slide pic goes, I’ve looked many times at this presentation.
It’s a bit confusing to me. The block labeled “client(s)”, are these the
datanode(s),
or are these really clients, like Spark clients? I was assuming these were the
Datanodes configured to be doing NVMf. But it was stated that the datanodes
aren’t involved with data movement, so it would have to be a like a spark
client, that is using the HDFS commands to the namenode. True?
And thanks for your patients… this is all reasonably new for us, but pretty
interesting!
Regards,
David
From: Jonas Pfefferle (JIRA)<mailto:[email protected]>
Sent: Wednesday, March 6, 2019 7:51 AM
To: David Crespi<mailto:[email protected]>
Subject: [jira] [Commented] (CRAIL-93) Using Crail with NVMf, the Default NQN
also attaches the port number to the name.
[
https://issues.apache.org/jira/browse/CRAIL-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785779#comment-16785779
]
Jonas Pfefferle commented on CRAIL-93:
--------------------------------------
Hi David,
# Take a look at
[https://www.openfabrics.org/images/2018workshop/presentations/111_BMetzler_NVMfLessons.pdf]
Slide 9. Maybe it makes it more clear. The namenode never is involved with
data IO only metadata operations.
I'm not saying it is not possible to have all subsystems on the same IP/port
pair in SPDK, the problem is in this case they have to have different subsystem
NQNs which is not supported by Crail since we only store IP/port information.
So the "solution" is to let every subsystem also have a different port, however
this still requires it to have a different subsystem NQN. Which led to the hack
of using the subsystem NQN with appended port numbers. Crail does not create
any subsystem NQNs (they are defined by the target and Crail does not have any
target capabilities since the underlying NVMf library only implements the
initiator side).
As you correctly stated above the host NQN and host ID identify the clients.
Currently the host NQN is created by Crail randomly and the host ID is set to
the MAC address of the first network interface found. I think we have a general
confusion about subsystem NQN and host NQN, this is how I understand the
specification: host NQN/host ID meant mainly for the clients to identify
themself and block/allow access on the target side and subsystem NQN on the
target side to identify a subsystem.
# I don't understand what you are saying here. The client always has to
specify the host NQN however the target can chose to not use it e.g. for access
control. I have been running SPDK like this with Crail without issues. Just
remove any "host" mention in the subsystem configuration and add "AllowAnyHost
Yes".
# I want to emphasize that the logging option is experimental thus it is not
extensively documented. Let us know if you find things in the documentation to
be confusing or incomplete, we try our best to add/fix them.
# Are you building on the provided Crail containers? How are you setting the
nqn in the Crail config?
Regards,
Jonas
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
> Using Crail with NVMf, the Default NQN also attaches the port number to the
> name.
> ---------------------------------------------------------------------------------
>
> Key: CRAIL-93
> URL: https://issues.apache.org/jira/browse/CRAIL-93
> Project: Apache Crail
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: I'm running crail nodes with docker containers, on a
> Ubuntu 18.04 base.
> Reporter: David Crespi
> Assignee: Jonas Pfefferle
> Priority: Critical
>
> This is the version I'm actually using: v1.1-2-gf0afadc
> I'm set up to use spdk on the backend to crail. When attempting to attach,
> it appears that crail needs two subsystems to achieve the connection. Instead
> of allowing the default name, I have the system variable set: -e
> NVMF_NQN="nqn.2017-06.io.crail:cnode"
> 1) subsystem NQN: nqn.2017-06.io.crail:cnode
> 2) subsystem NQN: nqn.2017-06.io.crail:cnode4420
> 19/03/01 17:46:45 INFO crail: CrailHadoopFileSystem fs initialization done..
> 19/03/01 17:46:45 INFO crail: Connecting to NVMf target at Transport address
> = /192.168.2.104:4420, subsystem NQN = nqn.2017-06.io.crail:cnode4420
> It appears that the initial connect/discovery of the subsystem uses #1, but
> using the
> crail commands (crail fs -mkdir /test) uses #2.
> Both have to have a valid namespace attached as well.
>
> It also appears that when using my own subsystem NQN (NVMF_NQN) name, crail
> wants to generate its own Host NQN. A new one every time. First, how do you
> learn of that
> NQN, and 2nd, it would be great to disable it if spdk has "allow any hosts"
> set. It refuses
> to connect to spdk.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
