This is an automated email from the ASF dual-hosted git repository.
nvazquez pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudstack.git
The following commit(s) were added to refs/heads/main by this push:
new e9af62c4ad5 flasharray/kvm/adaptive: NVMe-TCP transport for FlashArray
primary storage (#13061)
e9af62c4ad5 is described below
commit e9af62c4ad5d1b6af40dc059360821e367257ea8
Author: Eugenio Grosso <[email protected]>
AuthorDate: Mon Oct 26 12:32:19 2026 +0100
flasharray/kvm/adaptive: NVMe-TCP transport for FlashArray primary storage
(#13061)
* adapter: add NVMETCP address type and FlashArrayConnection.nsid
Preparatory data-model changes for NVMe-TCP support on the adaptive
storage framework. No behaviour change for existing Fibre Channel
users - the extra enum value, field, and getter/setter are only
exercised by callers that explicitly use them.
ProviderVolume.AddressType gains a NVMETCP value alongside FIBERWWN,
so adapters can declare that a volume is addressed by an NVMe EUI-128
(NGUID) rather than a SCSI WWN.
FlashArrayVolume.getAddress() produces the NGUID layout expected by
the Linux kernel for a FlashArray NVMe namespace:
00 + serial[0:14] + 24a937 (Pure 6-hex OUI) + serial[14:24]
which matches the /dev/disk/by-id/nvme-eui.<id> symlink emitted by
udev. Fibre Channel callers (addressType != NVMETCP) still get the
existing 6 + 24a9370 + serial form.
FlashArrayConnection gains a nsid field to carry the namespace id the
FlashArray REST API attaches to host-group-scoped NVMe connections,
when it is present.
* flasharray: support NVMe-TCP transport
Teach FlashArrayAdapter to talk to a pool over NVMe over TCP instead of
Fibre Channel.
The transport is selected from a new transport= option on the storage
pool URL (or the equivalent storage_pool_details entry), e.g.
https://user:pass@fa:443/api?pod=cs&transport=nvme-tcp&hostgroup=cluster1
Defaults remain Fibre Channel / WWN addressing when transport is absent
or anything other than nvme-tcp, so existing FC pools are unaffected.
Beyond the transport parsing itself the adapter now:
* Tracks a per-pool volumeAddressType (AddressType.NVMETCP or
FIBERWWN) and stamps every volume it hands back to the framework
with it (withAddressType), so the adaptive driver path stores the
correct type=... field in the CloudStack volume path (used later
by the KVM driver to locate the device).
* Attaches pod-backed NVMe-TCP volumes at the host-group level
(POST /connections?host_group_names=...) instead of per-host, so
the array assigns a consistent NSID to every member host; falls
back to per-host attach for FC or when no hostgroup is configured.
* Tolerates a missing nsid in the FlashArray connections response
for NVMe-TCP - Purity does not return one for host-group NVMe
connections; the namespace is identified on the host by EUI-128
from FlashArrayVolume.getAddress(), so a placeholder value is
returned to the caller purely for informational tracking.
* Resolves NVMETCP addresses back to volumes in getVolumeByAddress
by reversing the EUI-128 layout (strip optional eui. prefix, drop
leading 00 and the embedded Pure OUI).
* Indexes NVMe connections in getConnectionIdMap by host name (the
array returns one entry per host inside a host-group connection),
so connid.<hostname> tokens in the path still match in
parseAndValidatePath on the KVM side.
Followed by a matching adaptive/KVM driver change (separate commit).
* storage: add NVMeTCP storage pool type
NVMe-oF over TCP (NVMe-TCP) is conceptually a separate storage fabric
from Fibre Channel / iSCSI: it speaks the NVMe command set rather than
SCSI, identifies namespaces by EUI-128 NGUIDs rather than WWNs, and on
Linux is multipathed natively by the nvme driver rather than by
device-mapper multipath. Giving it its own StoragePoolType lets the
KVM agent dispatch the adaptive driver to a dedicated NVMe-oF adapter
(added in the next commit) without polluting the existing Fibre Channel
code path.
The new value is wired into the same format-routing and derivePath
fall-through paths that already special-case FiberChannel in
KVMStorageProcessor: NVMe-TCP volumes are also RAW and carry their
device path in DataObjectTO.path rather than in a managedStoreTarget
detail.
* kvm: add MultipathNVMeOFAdapterBase and NVMeTCPAdapter
Introduce an NVMe-over-Fabrics counterpart to the existing
MultipathSCSIAdapterBase / FiberChannelAdapter pair.
NVMe-oF is conceptually distinct from SCSI - it speaks the NVMe command
set, identifies namespaces by EUI-128 NGUIDs, and is multipathed by the
kernel natively rather than by device-mapper - so keeping it out of the
SCSI code path avoids special-casing inside every method that handles
volume paths, connect, disconnect, or size lookup.
MultipathNVMeOFAdapterBase (abstract)
* Parses volume paths of the form
type=NVMETCP; address=<eui>; connid.<host>=<nsid>; ...
into an AddressInfo whose path is
/dev/disk/by-id/nvme-eui.<eui>
which is the udev symlink the kernel emits for every NVMe namespace.
* connectPhysicalDisk polls the udev path and, on every iteration,
triggers nvme ns-rescan on all local NVMe controllers, to cover
target/firmware combinations that do not send an asynchronous event
notification when a new namespace is mapped.
* disconnectPhysicalDisk is a no-op; the kernel drops the namespace
when the target removes the host-group connection. The
ByPath variant only claims paths starting with
/dev/disk/by-id/nvme-eui. so foreign paths still fall through to
other adapters.
* Delegates getPhysicalDisk, isConnected, and getPhysicalDiskSize to
plain test -b / blockdev --getsize64 calls - no SCSI rescan, no dm
multipath, no multipath-map cleanup timer.
* createPhysicalDisk / createTemplateFromDisk / listPhysicalDisks /
copyPhysicalDisk all throw UnsupportedOperationException - these
are the responsibility of the storage provider, not the KVM
adapter, same as the SCSI base.
MultipathNVMeOFPool
* KVMStoragePool mirror of MultipathSCSIPool. Defaults to
Storage.StoragePoolType.NVMeTCP in the parameterless-fallback
constructor.
NVMeTCPAdapter
* Concrete adapter that registers itself for
Storage.StoragePoolType.NVMeTCP via the reflection-based scan in
KVMStoragePoolManager. Carries no logic of its own beyond binding
the base to the pool type.
A similar MultipathNVMeOFAdapterBase-derived NVMeRoCEAdapter (or
NVMeFCAdapter) can later be added by adding one concrete subclass and a
new pool-type value; the base does not assume any particular
fabric-level transport.
* adaptive: pick NVMeTCP pool type when transport=nvme-tcp
The adaptive storage framework hard-coded FiberChannel as the KVM-side
pool type for every provider it fronts. With a separate NVMeTCP pool
type now available (and a dedicated NVMe-oF adapter on the KVM side),
teach the lifecycle to route a pool to the right adapter based on a
transport= URL parameter:
https://user:pass@host/api?...&transport=nvme-tcp
-> StoragePoolType.NVMeTCP -> NVMeTCPAdapter on the KVM host
When the query parameter is absent the default stays FiberChannel, so
existing FC deployments on Primera or FlashArray continue to work
unchanged.
The choice is made in the shared AdaptiveDataStoreLifeCycleImpl rather
than inside each vendor plugin so every adaptive provider (FlashArray,
Primera, any future one) speaks the same configuration vocabulary.
* docs: note NVMe-TCP support on the FlashArray adaptive plugin in
PendingReleaseNotes
* kvm: implement copyPhysicalDisk on MultipathNVMeOFAdapterBase
The NVMe-oF KVM adapter refused every template copy request from the
adaptive storage orchestrator with UnsupportedOperationException, which
made it impossible to use an NVMe-TCP pool as primary storage for a VM
root disk: every deploy that landed a root volume on the pool failed
as soon as CloudStack tried to lay down the template.
Implement it the same way FiberChannel (SCSI) does: the storage provider
creates and connects a raw namespace ahead of time, then the adapter
resolves the host-side /dev/disk/by-id/nvme-eui.<NGUID> path via the
existing getPhysicalDisk plumbing (which will nvme ns-rescan and wait
for the symlink if the kernel has not yet picked it up) and qemu-img
converts the source image into the raw block device.
User-space encrypted source or destination volumes are rejected: the
FlashArray already encrypts at rest and layering qemu-img LUKS on top
of a hostgroup-scoped namespace shared between hosts is not a sensible
layering. Source encryption would also break on migration because the
passphrase does not travel.
With this change a CloudStack KVM VM can have its ROOT volume on an
NVMe-TCP pool (tested end-to-end on 4.23-SNAPSHOT against Purity 6.7.7:
template copy, first boot, live migrate with data disk, VM snapshot
with quiesce, and revert all work).
Signed-off-by: Eugenio Grosso <[email protected]>
* kvm/flasharray: address review feedback on NVMe-TCP PR
Apply the review comments from the first round on #13061:
* FlashArrayAdapter.snapshot() and both getSnapshot() entry points now
wrap the returned FlashArrayVolume in withAddressType(). Without this,
snapshots taken against an NVMe-TCP pool had the constructor-default
AddressType.FIBERWWN and ProviderSnapshot.getAddress() emitted an FC
style WWN instead of the NVMe EUI-128, which the adaptive driver then
persisted as the snapshot path. Verified end-to-end against Purity 6.7.7:
a fresh NVMe-TCP snapshot now lands with install_path starting 006c... ,
matching the source volume's EUI (previously it was 6-24a9370...).
* FlashArrayAdapter.attach() - retry path after 'Connection already
exists' no longer requires a hostgroup-scoped match for NVMe-TCP. If
hostgroup is not configured, or the existing connection is host-scoped,
fall back to matching by host name, same as the Fibre Channel branch.
Also normalize the 'volume lun is not found' message when no
connection list is returned.
* FlashArrayAdapter.attach() - initial 'Volume attach did not return lun
information' exception message now mentions both lun (FC) and nsid
(NVMe-TCP) so the error is not misleading on NVMe deployments.
* FlashArrayAdapter.getVolumeByAddress() - validate the EUI-128 length
before slicing. A short/malformed address used to throw
StringIndexOutOfBoundsException deep inside getFlashArrayItem and be
swallowed as 'not found'; now a clear RuntimeException is raised with
the expected vs actual length.
* FlashArrayVolume.getAddress() - same defensive check when building an
EUI-128 from the FlashArray volume serial; if the serial is shorter
than 24 hex chars, fail with a clear message instead of SIOOBE.
* MultipathNVMeOFAdapterBase.connectPhysicalDisk() - Integer.parseInt of
the STORAGE_POOL_DISK_WAIT detail is now guarded; a non-numeric value
falls back to the default rather than aborting the connect.
* MultipathNVMeOFAdapterBase.rescanAllControllers() - honour the boolean
return from Process.waitFor(). If an nvme ns-rescan invocation does
not complete in NS_RESCAN_TIMEOUT_SECS we destroyForcibly() it, so
hung nvme-cli processes do not accumulate while the namespace poll
loop retries.
* NVMeTCPAdapter - rename LOGGER_NVMETCP to LOGGER to match the naming
convention used in the other KVM adapters.
Signed-off-by: Eugenio Grosso <[email protected]>
* ui: expose NVMe-TCP transport for FlashArray primary storage
AddPrimaryStorage previously pinned the protocol to FiberChannel whenever
the operator picked the FlashArray provider, leaving NVMe-TCP backends only
reachable by hand-crafting the URL with ?transport=nvme-tcp. Surface the choice
in the form:
- protocols dropdown for FlashArray now offers FiberChannel and NVMeTCP
(Primera stays FC-only).
- when NVMeTCP is selected, the submit handler appends transport=nvme-tcp
to the FlashArray URL so the adaptive lifecycle pivot in pickPoolType()
resolves StoragePoolType.NVMeTCP server-side.
- the generic Path field, already hidden for FiberChannel, is also hidden
for NVMeTCP for parity.
Signed-off-by: Eugenio Grosso <[email protected]>
* kvm: align MultipathNVMeOFPool with KVMStoragePool API change
Reported by @slavkap: building this branch against current main produces a
compile error in MultipathNVMeOFPool because the KVMStoragePool interface
drifted between when this PR was last rebased (2026-05-22) and main today
(2026-06-09):
- checkingHeartBeat(HAStoragePool, HostTO) was renamed to
hasHeartBeat(HAStoragePool, HostTO)
- vmActivityCheck(...) was renamed to hasVmActivity(...) with the same
signature
Update the two overrides accordingly (both still return null as before —
MultipathNVMeOFPool does not participate in the KVM-side HA heartbeat). Mirrors
how MultipathSCSIPool implements them.
Verified compile via mvn -pl plugins/hypervisors/kvm -am -DskipTests
compile.
Signed-off-by: Eugenio Grosso <[email protected]>
* kvm: guard parseAndValidatePath against null inPath
@slavkap noted that parseAndValidatePath has no internal null guard. The
two callers in this base class (getPhysicalDisk and connectPhysicalDisk)
already pre-check via StringUtils.isEmpty(volumePath), so the current code path
is safe, but as a public method it deserves defense-in-depth.
Throw CloudRuntimeException with a clear message instead of letting it NPE
on inPath.split(...). Mirrors how the FC sibling FiberChannelAdapter throws on
an invalid address type.
Signed-off-by: Eugenio Grosso <[email protected]>
* kvm/flasharray/ui: address Copilot review round 3 on NVMe-TCP PR
Address the 2026-07-27 Copilot review comments:
- AddPrimaryStorage.vue: set (or replace) the transport= query parameter
instead of blindly appending it, so a FlashArray URL that already carries
a hand-entered transport= is not left with two conflicting values.
- FlashArrayVolume: slice the EUI-128 serial with explicit substring(14, 24)
rather than substring(14). The length guard rejected short serials but a
longer-than-expected serial would previously yield an EUI over 32 hex
characters, which cannot match /dev/disk/by-id/nvme-eui.<eui>.
- FlashArrayAdapter: guard the Fibre Channel connection match against
hostnames with no dot. hostname.substring(0, hostname.indexOf('.')) would
throw StringIndexOutOfBoundsException when indexOf('.') == -1. Mirrors the
guard already present on the NVMe-TCP branch.
- MultipathNVMeOFAdapterBase: MapStorageUuidToStoragePool is now a
ConcurrentHashMap populated via computeIfAbsent, so concurrent callers
cannot race to create duplicate pool objects or corrupt the map.
- MultipathNVMeOFAdapterBase: split key=value tokens with a limit of 2 so a
value containing '=' is no longer silently discarded.
- MultipathNVMeOFAdapterBase: throttle rescanAllControllers() during
namespace discovery to once per 10s instead of on every 2s poll. Spawning
one nvme ns-rescan per controller every 2s is bursty on hosts with many
controllers; the first iteration still rescans immediately.
No functional change to the happy path.
Signed-off-by: Eugenio Grosso <[email protected]>
* flasharray: tighten NVMe-TCP EUI-128 validation
Address the 2026-08-04 Copilot follow-up:
- FlashArrayVolume.getAddress() now requires the serial to match exactly 24
hexadecimal characters instead of merely rejecting shorter ones. The
previous
form sliced serial[0:24], so two distinct serials sharing a 24-character
prefix would have produced the same EUI-128 and therefore the same volume
identity. Non-hex serials are now rejected as well.
- FlashArrayAdapter.getVolumeByAddress() validates the FlashArray EUI-128
layout before reversing it into a serial: 32 hexadecimal characters, a 00
prefix, and the Pure Storage OUI at offset 16. Previously any 32-character
string was accepted and deterministically mapped onto a volume serial, so
a
malformed or tampered address could resolve to an unintended volume.
Verified against a real FlashArray namespace: EUI
006c1b16ce1c034d24a9371c05ab334a passes both checks and round-trips to
serial
6C1B16CE1C034D1C05AB334A and back unchanged.
Signed-off-by: Eugenio Grosso <[email protected]>
* kvm: implement volume resize for NVMe-TCP pools
Resize of a volume on an NVMe-TCP pool previously failed. The provider half
worked -- FlashArrayAdapter.resize() PATCHes the array to grow the
namespace --
but the KVM half was never wired up:
* LibvirtResizeVolumeCommandWrapper short-circuits MultipathSCSIPool via
handleMultipathSCSIResize(), but MultipathNVMeOFPool implements
KVMStoragePool directly rather than extending MultipathSCSIPool, so it
fell through.
* It then reached LibvirtComputingResource.getResizeScriptType(), which
has
no NVMeTCP case, and threw
"Cannot determine resize type from pool type NVMeTCP".
* MultipathNVMeOFAdapterBase.resize() threw UnsupportedOperationException.
Wire up the host-side half, mirroring how the Fibre Channel path is handled
(FC shipped resize in its own initial PR, #7889):
* LibvirtResizeVolumeCommandWrapper: dispatch MultipathNVMeOFPool to a new
handleMultipathNVMeOFResize(), alongside the existing SCSI branch.
* MultipathNVMeOFPool.resize(): delegate to the adaptor, mirroring
MultipathSCSIPool.resize().
* MultipathNVMeOFAdapterBase.resize(): issue nvme ns-rescan so the kernel
observes the grown namespace without waiting for an asynchronous event
notification from the target, poll until the new size is visible, then
virsh blockresize so a running guest sees the capacity without a reboot.
Unlike the SCSI path there is no device-mapper map to grow, so no helper
script is needed; the existing rescanAllControllers() is reused.
If the namespace does not report the new size within the settle timeout we
log a warning rather than throwing: the array has already been grown at that
point, and failing here would make the management server roll back a resize
that actually happened.
The guest-notification lookup matches the domain disk source against the
/dev/disk/by-id symlink, its canonicalised /dev/nvmeXnY target, or the bare
EUI, because libvirt may report any of those forms.
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* server: allow online volume resize on NVMe-TCP pools
validateVolumeResizeWithSize() refuses to resize a volume on a managed pool
while its VM is running, unless the pool type is on an allowlist. That list
held PowerFlex and FiberChannel but not NVMeTCP, so resizing an attached
NVMe-TCP volume on a running KVM instance failed at the API layer with
"This kind of KVM disk cannot be resized while it is connected to a VM
that's not in the Stopped state." before any host-side work was attempted.
Add NVMeTCP alongside FiberChannel. The two transports are equivalent here:
in both cases the provider grows the volume on the array and the KVM agent
then makes the new capacity visible to the guest, which the preceding commit
implements for NVMe-oF.
Found by running the test matrix @abh1sar requested on
apache/cloudstack#13061:
offline resize (VM Stopped) already worked end to end -- CloudStack, the
array
and the guest all agreed on the new size -- and this guard was the only
thing
blocking the online case.
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* storage: recognise NVMeTCP in the managed-storage code paths
When Fibre Channel managed storage was added (#7889) the FiberChannel pool
type was added to a number of allowlists in shared managed-storage code. The
NVMe-TCP work added the pool type, the KVM adaptor and the lifecycle pivot
but
not those allowlists, so several managed-storage operations silently did not
work on an NVMe-TCP pool even though the equivalent Fibre Channel operation
did.
StorageSystemDataMotionStrategy:
* verifyFormatWithPoolType(): RAW was accepted only on PowerFlex and
FiberChannel, so any data-motion of a RAW NVMe-TCP volume was rejected
outright. Message updated to match.
* handleCreateTemplateFromManagedVolume(): same format check, which made
createTemplate from an NVMe-TCP volume fail with the (misleading)
"When using managed storage, you can only create a template from a
volume
on KVM currently."
* the grantAccess() / revokeAccess() pair around the template copy only
ran
for a detached volume or for PowerFlex/FiberChannel, so an attached
NVMe-TCP volume was never granted host access for the copy. Both sites
are
updated together so access is still revoked in the finally block.
* getSnapshotDetails(): PowerFlex and FiberChannel pass the snapshot path
straight through as the device address, which is also correct for
array-native NVMe-TCP snapshots.
ApiDBUtils.getHypervisorTypeFromZone(): NVMeTCP is a KVM-only pool type, so
its presence in a zone should imply KVM in the same way the other non-QCOW2
pool types already do.
StatsCollector's volume-stats format check deliberately not touched: RAW is
already in its accepted-format list, so the guard never fires for NVMe-TCP
volumes and adding the pool type there would be dead weight.
Found by working through the test matrix @abh1sar asked for on
apache/cloudstack#13061, which surfaced these as a family rather than one at
a time.
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* flasharray: connect NVMe-TCP volumes per host, not per host group
Access to a managed volume is granted and revoked one host at a time:
AdaptiveDataStoreDriverImpl.grantAccess() calls attach(volume, hostname) and
revokeAccess() calls detach(volume, hostname). Neither the adapter context
nor
the data object carries the VM or its power state, so the adapter cannot
tell
why access is being revoked.
attach() did not follow that model for NVMe-TCP. When a hostgroup was
configured it created a single host-group scoped connection shared by every
member host, while detach() still ran per host and deleted that shared
connection. The two are irreconcilable, and live migration shows why:
grantAccess(V, dst) -> POST /connections?host_group_names=cluster1
("already exists")
... guest migrates and is now running on dst, doing I/O to V ...
revokeAccess(V, src) -> DELETE /connections?host_group_names=cluster1
The DELETE removes the only connection there is, so V is revoked from the
whole
group including dst, the namespace disappears under the running guest, and
with
werror=stop the VM freezes.
Group scoping was justified in a comment as giving "a consistent NSID
visible to
every member host", but the NSID is not used to find the namespace: it is
located by its EUI-128 address (FlashArrayVolume.getAddress()), as the
comment
20 lines further down already says. So the justification does not hold.
Connect and disconnect per host instead, which is what Fibre Channel has
always
done in this same adapter and what the interface can express. The hostgroup
parameter remains accepted and is now used only where it was documented to
be
used -- removing leftover group connections when a volume is deleted --
matching
the "retrieve for legacy purposes" comment on the field. In the "Connection
already exists" path a group-scoped match is demoted to a fallback so
volumes
still carrying a group connection from an earlier release keep resolving.
Found while working through the test matrix @abh1sar asked for on
apache/cloudstack#13061, as the cause of the live-migration failure.
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* flasharray: always stamp the pool address type on returned volumes
FlashArrayVolume defaults addressType to FIBERWWN in its constructor, so a
volume deserialised from the array reports an FC-style NAA WWN from
getAddress()
until the pool's real address type is stamped onto it. Most accessors do
that
via withAddressType(), but two paths returned an unstamped volume:
* revert() returned the private getVolume() result directly, so reverting
a
snapshot on an NVMe-TCP pool produced a volume carrying an FC-style WWN.
* copy() returned the newly created volume unstamped. The adaptive driver
persists this one -- copyAsync() stores generatePathInfo(outVolume,
...) --
so the volume was recorded in the database with an address the host
cannot
resolve, since NVMe namespaces are located by EUI-128.
Rather than add a third withAddressType() call at each site, stamp inside
the
private getVolume() accessor so every path is correct by construction. That
matches the private getSnapshot() accessor, which already did this, and
makes
the wrap at the public getVolume() call site redundant. copy() now reuses
the
accessor for its source lookup instead of repeating the same GET inline, and
stamps the volume it hands back.
No behavioural change for Fibre Channel: FIBERWWN is what the constructor
already defaulted to, so stamping it is a no-op there.
Also drops a comment in getConnectionIdMap() describing connections as
host-group scoped, which is no longer how NVMe-TCP volumes are connected
after
the preceding commit. The surrounding logic was already keyed on host name
and
is unchanged; only nsid vs lun differs between the transports.
Found by auditing every NVMe-TCP divergence from the Fibre Channel path, on
the
principle that NVMe-TCP is only a different carrier: the array operations
are
identical and the host-side difference is an NQN instead of a WWN, which
lives
on the array and is never touched by this adapter (hosts are resolved by
name).
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* kvm: align NVMe-oF adaptor stubs with the Fibre Channel adaptor
NVMe-TCP is a different carrier for the same array operations, so the
NVMe-oF
storage adaptor should only diverge from MultipathSCSIAdapterBase where the
transport genuinely differs. Several unimplemented methods threw
UnsupportedOperationException where the Fibre Channel adaptor returns a
benign
value, which means a caller that degrades gracefully on Fibre Channel
aborts on
NVMe-TCP:
* deletePhysicalDisk(): FC returns false. Namespaces are created and
destroyed
by the storage provider, so a host-side delete is simply not handled
here;
throwing from a cleanup path is worse than reporting that.
* createTemplateFromDisk(), listPhysicalDisks(): FC returns null.
* createFolder(): FC logs and returns true. Block storage has no directory
structure, so this is a no-op rather than an error.
createPhysicalDisk(), createDiskFromTemplate() and
createDiskFromTemplateBacking()
still throw, because the Fibre Channel adaptor throws for those too.
createTemplateFromDirectDownloadFile() also still throws, but the message
now
states the limitation instead of reading as an oversight. The Fibre Channel
implementation resolves the downloaded file with destPool.getPhysicalDisk(),
which on this adaptor requires a "type=NVMETCP;address=..." path and
therefore
cannot describe a plain local file. Supporting it needs a separate way to
present
the local file as the copy source, which is left for follow-up rather than
guessed at here.
No functional change to the connect/disconnect paths. The comment on
disconnectPhysicalDisk() is updated because it described connections as
host-group scoped, which is no longer how NVMe-TCP volumes are connected,
and to
record why the host-side no-op is correct: the kernel drops the namespace
when
the target removes the connection, whereas Fibre Channel must flush its
device-mapper entry.
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
* kvm: support direct download templates on NVMe-oF pools
createTemplateFromDirectDownloadFile() threw UnsupportedOperationException,
so a
template registered with directdownload=true could not be deployed onto an
NVMe-TCP pool: the download itself succeeded and the deployment then failed
on
the host.
The two path arguments are different kinds of thing, and conflating them is
what
made this look harder than it is. templateFilePath is a plain local file
produced
by the direct-download helper, while destTemplatePath is a managed volume
path of
the form "type=NVMETCP;address=...". Only the destination may be resolved
through
KVMStoragePool.getPhysicalDisk(); handing the local file to that would pass
a file
name to parseAndValidatePath() and fail. KVMStorageProcessor has already
issued
connectPhysicalDisk() for the destination by the time we are called, so the
namespace is present.
Worth recording that the Fibre Channel implementation does conflate them.
It calls
destPool.getPhysicalDisk(templateFilePath), and FiberChannelAdapter's
parseAndValidatePath() treats any string without a ";" as a bare WWN, so a
local
file path becomes "/dev/mapper/3" + that path and the lookup cannot
succeed. This
implementation follows ScaleIOStorageAdaptor instead, which is the working
precedent for block-backed managed storage.
The template is written as QCOW2 onto the raw namespace rather than as RAW.
Three
reasons: it matches ScaleIO; it agrees with the QCOW2 format the template is
registered with, so stored content and declared format do not contradict
each
other; and KVMStorageProcessor runs Qcow2Inspector.validateQcow2File() on
the path
we return, which for a block device that exists would fail on RAW content
and then
try to delete the device.
The disk returned to the caller is constructed rather than being the one
getPhysicalDisk() handed back, because its *name* matters and not just its
path.
KVMStorageProcessor puts disk.getName() into the DirectDownloadAnswer, which
becomes the template's install path in template_spool_ref and later its
external
name, and the provider interpolates that external name into array REST
calls.
getPhysicalDisk() names a disk AddressInfo.toString(), e.g.
AddressInfo NVMETCP [address=006c..., connectionId=2172,
path=/dev/disk/by-id/...]
Recording that as the install path makes the first deployment from a freshly
registered template fail: revokeAccess() -> detach() builds
"/connections?host_names=kvm01&volume_names=cloudstack2::<external name>"
and
URI.create() rejects the spaces and brackets with "Illegal character in
query".
The surfaced error is a misleading "Failed to update state", because the
resulting
CloudRuntimeException is followed by applying OperationFailed to a template
that
has already reached Ready. Naming the returned disk with the managed volume
path
gives the same install path shape a volume records:
type=NVMETCP; address=006c...; providerName=cloudstack2::tpl-9-0-2-210;
providerID=...
Also handles compressed templates via TemplateDownloaderUtil, since
direct-download templates are commonly published that way, and refuses up
front if
the template will not fit the namespace rather than failing part way
through the
convert.
Tested on a two-host KVM cluster against a FlashArray over NVMe-TCP:
registered a
QCOW2 template with directdownload=true and deployed a VM from it, on the
first
attempt, with a storage offering tagged for the NVMe-TCP pool:
ROOT-32 Everpure X90 NVMe-TCP pool2 Ready
type=NVMETCP; address=006c1b16ce1c034d24a9371c0623b125
Signed-off-by: Eugenio Grosso <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
---------
Signed-off-by: Eugenio Grosso <[email protected]>
Co-authored-by: Eugenio Grosso <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
---
PendingReleaseNotes | 10 +
api/src/main/java/com/cloud/storage/Storage.java | 3 +-
.../motion/StorageSystemDataMotionStrategy.java | 17 +-
.../wrapper/LibvirtResizeVolumeCommandWrapper.java | 10 +
.../kvm/storage/KVMStorageProcessor.java | 9 +-
.../kvm/storage/MultipathNVMeOFAdapterBase.java | 722 +++++++++++++++++++++
.../kvm/storage/MultipathNVMeOFPool.java | 165 +++++
.../hypervisor/kvm/storage/NVMeTCPAdapter.java | 51 ++
.../storage/datastore/adapter/ProviderVolume.java | 3 +-
.../lifecycle/AdaptiveDataStoreLifeCycleImpl.java | 24 +-
.../adapter/flasharray/FlashArrayAdapter.java | 160 ++++-
.../adapter/flasharray/FlashArrayConnection.java | 9 +
.../adapter/flasharray/FlashArrayVolume.java | 26 +
server/src/main/java/com/cloud/api/ApiDBUtils.java | 3 +-
.../com/cloud/storage/VolumeApiServiceImpl.java | 3 +-
ui/src/views/infra/AddPrimaryStorage.vue | 17 +-
16 files changed, 1181 insertions(+), 51 deletions(-)
diff --git a/PendingReleaseNotes b/PendingReleaseNotes
index bc6068fa331..c9afb25f7ed 100644
--- a/PendingReleaseNotes
+++ b/PendingReleaseNotes
@@ -81,3 +81,13 @@ example.ver.1 > example.ver.2:
additional key store is required. Note: attaching an encrypted RBD volume
to a running Instance requires libvirt >= 10.1.0; booting an Instance from
an encrypted RBD root disk works on older libvirt.
+
+ * Added NVMe-over-Fabrics (TCP) support to the adaptive storage framework
+ and the Pure Storage FlashArray plugin. Volumes on a FlashArray primary
+ pool can now be delivered to KVM hypervisors over NVMe-TCP instead of
+ Fibre Channel by setting transport=nvme-tcp on the pool's provider URL.
+ Volumes are identified on the host via EUI-128 NGUIDs and attached to
+ guests as plain block devices through the native NVMe multipath layer;
+ no device-mapper multipath configuration is required. A new
+ Storage.StoragePoolType.NVMeTCP + MultipathNVMeOFAdapterBase /
+ NVMeTCPAdapter on the KVM side back the new pool type.
diff --git a/api/src/main/java/com/cloud/storage/Storage.java
b/api/src/main/java/com/cloud/storage/Storage.java
index 275f6d25268..42a18b9c83c 100644
--- a/api/src/main/java/com/cloud/storage/Storage.java
+++ b/api/src/main/java/com/cloud/storage/Storage.java
@@ -185,7 +185,8 @@ public class Storage {
Linstor(true, true, EncryptionSupport.Storage),
DatastoreCluster(true, true, EncryptionSupport.Unsupported), // for
VMware, to abstract pool of clusters
StorPool(true, true, EncryptionSupport.Hypervisor),
- FiberChannel(true, true, EncryptionSupport.Unsupported); // Fiber
Channel Pool for KVM hypervisors is used to find the volume by WWN value
(/dev/disk/by-id/wwn-<wwnvalue>)
+ FiberChannel(true, true, EncryptionSupport.Unsupported), // Fiber
Channel Pool for KVM hypervisors is used to find the volume by WWN value
(/dev/disk/by-id/wwn-<wwnvalue>)
+ NVMeTCP(true, true, EncryptionSupport.Unsupported); // NVMe over TCP
(NVMe-oF/TCP) Pool for KVM hypervisors; volumes are identified by EUI-128 NGUID
(/dev/disk/by-id/nvme-eui.<eui>)
private final boolean shared;
private final boolean overProvisioning;
diff --git
a/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java
b/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java
index 7674f1ce25a..2a348cb058d 100644
---
a/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java
+++
b/engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/StorageSystemDataMotionStrategy.java
@@ -616,8 +616,9 @@ public class StorageSystemDataMotionStrategy implements
DataMotionStrategy {
private void verifyFormatWithPoolType(ImageFormat imageFormat,
StoragePoolType poolType) {
if (imageFormat != ImageFormat.VHD && imageFormat != ImageFormat.OVA
&& imageFormat != ImageFormat.QCOW2 &&
!(imageFormat == ImageFormat.RAW && (StoragePoolType.PowerFlex
== poolType ||
- StoragePoolType.FiberChannel == poolType))) {
- throw new CloudRuntimeException(String.format("Only the following
image types are currently supported: %s, %s, %s, %s (for PowerFlex and
FiberChannel)",
+ StoragePoolType.FiberChannel == poolType ||
+ StoragePoolType.NVMeTCP == poolType))) {
+ throw new CloudRuntimeException(String.format("Only the following
image types are currently supported: %s, %s, %s, %s (for PowerFlex,
FiberChannel and NVMeTCP)",
ImageFormat.VHD.toString(), ImageFormat.OVA.toString(),
ImageFormat.QCOW2.toString(), ImageFormat.RAW.toString()));
}
}
@@ -2796,7 +2797,8 @@ public class StorageSystemDataMotionStrategy implements
DataMotionStrategy {
if (!ImageFormat.QCOW2.equals(volumeInfo.getFormat()) &&
!(ImageFormat.RAW.equals(volumeInfo.getFormat()) && (
StoragePoolType.PowerFlex == storagePoolVO.getPoolType() ||
- StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType()))) {
+ StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType() ||
+ StoragePoolType.NVMeTCP == storagePoolVO.getPoolType()))) {
throw new CloudRuntimeException("When using managed storage,
you can only create a template from a volume on KVM currently.");
}
@@ -2810,7 +2812,8 @@ public class StorageSystemDataMotionStrategy implements
DataMotionStrategy {
try {
handleQualityOfServiceForVolumeMigration(volumeInfo,
PrimaryDataStoreDriver.QualityOfServiceState.MIGRATION);
- if (srcVolumeDetached || StoragePoolType.PowerFlex ==
storagePoolVO.getPoolType() || StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType()) {
+ if (srcVolumeDetached || StoragePoolType.PowerFlex ==
storagePoolVO.getPoolType() || StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType()
+ || StoragePoolType.NVMeTCP ==
storagePoolVO.getPoolType()) {
_volumeService.grantAccess(volumeInfo, hostVO,
srcDataStore);
}
@@ -2844,7 +2847,8 @@ public class StorageSystemDataMotionStrategy implements
DataMotionStrategy {
throw new CloudRuntimeException(msg + ex.getMessage(), ex);
}
finally {
- if (srcVolumeDetached || StoragePoolType.PowerFlex ==
storagePoolVO.getPoolType() || StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType()) {
+ if (srcVolumeDetached || StoragePoolType.PowerFlex ==
storagePoolVO.getPoolType() || StoragePoolType.FiberChannel ==
storagePoolVO.getPoolType()
+ || StoragePoolType.NVMeTCP ==
storagePoolVO.getPoolType()) {
try {
_volumeService.revokeAccess(volumeInfo, hostVO,
srcDataStore);
}
@@ -2939,7 +2943,8 @@ public class StorageSystemDataMotionStrategy implements
DataMotionStrategy {
long snapshotId = snapshotInfo.getId();
- if (storagePoolVO.getPoolType() == StoragePoolType.PowerFlex ||
storagePoolVO.getPoolType() == StoragePoolType.FiberChannel) {
+ if (storagePoolVO.getPoolType() == StoragePoolType.PowerFlex ||
storagePoolVO.getPoolType() == StoragePoolType.FiberChannel
+ || storagePoolVO.getPoolType() == StoragePoolType.NVMeTCP) {
snapshotDetails.put(DiskTO.IQN, snapshotInfo.getPath());
} else {
snapshotDetails.put(DiskTO.IQN, getSnapshotProperty(snapshotId,
DiskTO.IQN));
diff --git
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java
index 20e3891478b..32e756607e5 100644
---
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java
+++
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtResizeVolumeCommandWrapper.java
@@ -48,6 +48,7 @@ import
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource;
import com.cloud.hypervisor.kvm.storage.KVMPhysicalDisk;
import com.cloud.hypervisor.kvm.storage.KVMStoragePool;
import com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager;
+import com.cloud.hypervisor.kvm.storage.MultipathNVMeOFPool;
import com.cloud.hypervisor.kvm.storage.MultipathSCSIPool;
import com.cloud.resource.CommandWrapper;
import com.cloud.resource.ResourceWrapper;
@@ -86,6 +87,10 @@ public final class LibvirtResizeVolumeCommandWrapper extends
CommandWrapper<Resi
return handleMultipathSCSIResize(command, pool);
}
+ if (pool instanceof MultipathNVMeOFPool) {
+ return handleMultipathNVMeOFResize(command, pool);
+ }
+
if (spool.getType().equals(StoragePoolType.PowerFlex)) {
pool.connectPhysicalDisk(volumeId, null);
}
@@ -239,4 +244,9 @@ public final class LibvirtResizeVolumeCommandWrapper
extends CommandWrapper<Resi
((MultipathSCSIPool)pool).resize(command.getPath(),
command.getInstanceName(), command.getNewSize());
return new ResizeVolumeAnswer(command, true, "");
}
+
+ private Answer handleMultipathNVMeOFResize(ResizeVolumeCommand command,
KVMStoragePool pool) {
+ ((MultipathNVMeOFPool)pool).resize(command.getPath(),
command.getInstanceName(), command.getNewSize());
+ return new ResizeVolumeAnswer(command, true, "");
+ }
}
diff --git
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
index 0cb32da318e..29b23e6c645 100644
---
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
+++
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java
@@ -421,7 +421,8 @@ public class KVMStorageProcessor implements
StorageProcessor {
StoragePoolType.PowerFlex,
StoragePoolType.Linstor,
StoragePoolType.FiberChannel,
- StoragePoolType.CLVM).contains(primaryPool.getType())) {
+ StoragePoolType.CLVM,
+ StoragePoolType.NVMeTCP).contains(primaryPool.getType())) {
newTemplate.setFormat(ImageFormat.RAW);
} else {
newTemplate.setFormat(ImageFormat.QCOW2);
@@ -454,7 +455,8 @@ public class KVMStorageProcessor implements
StorageProcessor {
public static String derivePath(PrimaryDataStoreTO primaryStore, DataTO
destData, Map<String, String> details) {
String path = null;
- if (primaryStore.getPoolType() == StoragePoolType.FiberChannel) {
+ if (primaryStore.getPoolType() == StoragePoolType.FiberChannel
+ || primaryStore.getPoolType() == StoragePoolType.NVMeTCP) {
path = destData.getPath();
} else {
path = details != null ? details.get("managedStoreTarget") : null;
@@ -3513,7 +3515,8 @@ public class KVMStorageProcessor implements
StorageProcessor {
StoragePoolType.PowerFlex,
StoragePoolType.Linstor,
StoragePoolType.FiberChannel,
- StoragePoolType.CLVM).contains(poolType)) {
+ StoragePoolType.CLVM,
+ StoragePoolType.NVMeTCP).contains(poolType)) {
return ImageFormat.RAW;
} else {
return ImageFormat.QCOW2;
diff --git
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFAdapterBase.java
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFAdapterBase.java
new file mode 100644
index 00000000000..2128d96ccf1
--- /dev/null
+++
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFAdapterBase.java
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package com.cloud.hypervisor.kvm.storage;
+
+import java.io.File;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImg.PhysicalDiskFormat;
+import org.apache.cloudstack.utils.qemu.QemuImgException;
+import org.apache.cloudstack.utils.qemu.QemuImgFile;
+import org.libvirt.LibvirtException;
+
+import com.cloud.storage.Storage;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.OutputInterpreter;
+import com.cloud.utils.script.Script;
+import com.cloud.utils.storage.TemplateDownloaderUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * Base class for KVM storage adapters that surface remote block volumes over
+ * NVMe-over-Fabrics (NVMe-oF). It is the NVMe-oF counterpart of
+ * {@link MultipathSCSIAdapterBase}: it does not drive device-mapper multipath
+ * and does not rescan the SCSI bus, because NVMe-oF has its own multipath
+ * (the kernel's native NVMe multipath) and namespaces show up via
+ * asynchronous event notifications as soon as the target grants access.
+ *
+ * Volumes are identified on the host by their EUI-128 NGUID, which udev
+ * exposes as {@code /dev/disk/by-id/nvme-eui.<eui>}.
+ */
+public abstract class MultipathNVMeOFAdapterBase implements StorageAdaptor {
+ protected static Logger LOGGER =
LogManager.getLogger(MultipathNVMeOFAdapterBase.class);
+ static final Map<String, KVMStoragePool> MapStorageUuidToStoragePool = new
ConcurrentHashMap<>();
+
+ static final int DEFAULT_DISK_WAIT_SECS = 240;
+ static final long NS_RESCAN_TIMEOUT_SECS = 5;
+ private static final long POLL_INTERVAL_MS = 2000;
+ private static final long RESCAN_INTERVAL_MS = 10_000;
+ private static final long RESIZE_SETTLE_TIMEOUT_MS = 30_000;
+
+ @Override
+ public KVMStoragePool getStoragePool(String uuid) {
+ // Dummy pool - adapters that dispatch per-volume don't need
+ // connectivity information on the pool itself. Use computeIfAbsent
+ // so concurrent callers do not race to create duplicate pool objects.
+ return MapStorageUuidToStoragePool.computeIfAbsent(uuid, u -> new
MultipathNVMeOFPool(u, this));
+ }
+
+ @Override
+ public KVMStoragePool getStoragePool(String uuid, boolean refreshInfo) {
+ return getStoragePool(uuid);
+ }
+
+ public abstract String getName();
+
+ @Override
+ public abstract Storage.StoragePoolType getStoragePoolType();
+
+ public abstract boolean isStoragePoolTypeSupported(Storage.StoragePoolType
type);
+
+ /**
+ * Parse a {@code type=NVMETCP; address=<eui>; connid.<host>=<nsid>; ...}
+ * volume path and produce an {@link AddressInfo} with the host-side device
+ * path set to {@code /dev/disk/by-id/nvme-eui.<eui>}.
+ */
+ public AddressInfo parseAndValidatePath(String inPath) {
+ if (inPath == null) {
+ throw new CloudRuntimeException("Cannot parse null volume path");
+ }
+ String type = null;
+ String address = null;
+ String connectionId = null;
+ String path = null;
+ String hostname = resolveHostnameShort();
+ String hostnameFq = resolveHostnameFq();
+ String[] parts = inPath.split(";");
+ for (String part : parts) {
+ // Cap the split at 2 so values containing '=' (e.g. base64) are
not silently discarded.
+ String[] pair = part.split("=", 2);
+ if (pair.length != 2) {
+ continue;
+ }
+ String key = pair[0].trim();
+ String value = pair[1].trim();
+ if (key.equals("type")) {
+ type = value.toUpperCase();
+ } else if (key.equals("address")) {
+ address = value;
+ } else if (key.equals("connid")) {
+ connectionId = value;
+ } else if (key.startsWith("connid.")) {
+ String inHostname = key.substring("connid.".length());
+ if (inHostname.equals(hostname) ||
inHostname.equals(hostnameFq)) {
+ connectionId = value;
+ }
+ }
+ }
+
+ if (!"NVMETCP".equals(type)) {
+ throw new CloudRuntimeException("Invalid address type provided for
NVMe-oF target disk: " + type);
+ }
+ if (address == null) {
+ throw new CloudRuntimeException("NVMe-oF volume path is missing
the required address field");
+ }
+ path = "/dev/disk/by-id/nvme-eui." + address.toLowerCase();
+ return new AddressInfo(type, address, connectionId, path);
+ }
+
+ @Override
+ public KVMPhysicalDisk getPhysicalDisk(String volumePath, KVMStoragePool
pool) {
+ if (StringUtils.isEmpty(volumePath) || pool == null) {
+ LOGGER.error("Unable to get physical disk, volume path or pool not
specified");
+ return null;
+ }
+ return getPhysicalDisk(parseAndValidatePath(volumePath), pool);
+ }
+
+ private KVMPhysicalDisk getPhysicalDisk(AddressInfo address,
KVMStoragePool pool) {
+ KVMPhysicalDisk disk = new KVMPhysicalDisk(address.getPath(),
address.toString(), pool);
+ disk.setFormat(QemuImg.PhysicalDiskFormat.RAW);
+
+ if (!isConnected(address.getPath())) {
+ if (!connectPhysicalDisk(address, pool, null)) {
+ throw new CloudRuntimeException("Unable to connect to NVMe
namespace at " + address.getPath());
+ }
+ }
+ long diskSize = getPhysicalDiskSize(address.getPath());
+ disk.setSize(diskSize);
+ disk.setVirtualSize(diskSize);
+ return disk;
+ }
+
+ @Override
+ public KVMStoragePool createStoragePool(String uuid, String host, int
port, String path, String userInfo, Storage.StoragePoolType type, Map<String,
String> details, boolean isPrimaryStorage) {
+ LOGGER.info(String.format("createStoragePool(uuid,host,port,path,type)
called with args (%s, %s, %d, %s, %s)", uuid, host, port, path, type));
+ MultipathNVMeOFPool pool = new MultipathNVMeOFPool(uuid, host, port,
path, type, details, this);
+ MapStorageUuidToStoragePool.put(uuid, pool);
+ return pool;
+ }
+
+ @Override
+ public boolean deleteStoragePool(String uuid) {
+ MapStorageUuidToStoragePool.remove(uuid);
+ return true;
+ }
+
+ @Override
+ public boolean deleteStoragePool(KVMStoragePool pool) {
+ return deleteStoragePool(pool.getUuid());
+ }
+
+ @Override
+ public boolean connectPhysicalDisk(String volumePath, KVMStoragePool pool,
Map<String, String> details, boolean isVMMigrate) {
+ if (StringUtils.isEmpty(volumePath) || pool == null) {
+ LOGGER.error("Unable to connect NVMe-oF physical disk:
insufficient arguments");
+ return false;
+ }
+ return connectPhysicalDisk(parseAndValidatePath(volumePath), pool,
details);
+ }
+
+ private boolean connectPhysicalDisk(AddressInfo address, KVMStoragePool
pool, Map<String, String> details) {
+ if (address.getConnectionId() == null) {
+ LOGGER.error("NVMe-oF volume " + address.getPath() + " on pool " +
pool.getUuid() + " is missing a connid.<host> token in its path");
+ return false;
+ }
+ long waitSecs = DEFAULT_DISK_WAIT_SECS;
+ if (details != null &&
details.containsKey(com.cloud.storage.StorageManager.STORAGE_POOL_DISK_WAIT.toString()))
{
+ String waitTime =
details.get(com.cloud.storage.StorageManager.STORAGE_POOL_DISK_WAIT.toString());
+ if (StringUtils.isNotEmpty(waitTime)) {
+ try {
+ waitSecs = Integer.parseInt(waitTime);
+ } catch (NumberFormatException e) {
+ LOGGER.warn("Ignoring non-numeric " +
com.cloud.storage.StorageManager.STORAGE_POOL_DISK_WAIT.toString()
+ + "=[" + waitTime + "] on pool " + pool.getUuid()
+ ", falling back to default "
+ + DEFAULT_DISK_WAIT_SECS + "s");
+ }
+ }
+ }
+ return waitForNamespace(address, pool, waitSecs);
+ }
+
+ /**
+ * Poll for the EUI-keyed udev symlink to show up. On every iteration also
+ * nudge the kernel with {@code nvme ns-rescan} on every local NVMe
+ * controller, to cover arrays / firmware combinations that do not emit a
+ * reliable asynchronous event notification when a new namespace is
+ * mapped.
+ */
+ private boolean waitForNamespace(AddressInfo address, KVMStoragePool pool,
long waitSecs) {
+ if (waitSecs < 60) {
+ waitSecs = 60;
+ }
+ long deadline = System.currentTimeMillis() + (waitSecs * 1000);
+ File dev = new File(address.getPath());
+ long lastRescan = 0;
+ while (System.currentTimeMillis() < deadline) {
+ if (dev.exists() && isConnected(address.getPath())) {
+ long size = getPhysicalDiskSize(address.getPath());
+ if (size > 0) {
+ LOGGER.debug("Found NVMe namespace at " +
address.getPath());
+ return true;
+ }
+ }
+ // Throttle rescanAllControllers(): spawning one nvme ns-rescan
per controller every
+ // 2s can be expensive on hosts with many controllers.
RESCAN_INTERVAL_MS caps the
+ // rate; the first iteration still rescans immediately because
lastRescan starts at 0.
+ long now = System.currentTimeMillis();
+ if (now - lastRescan >= RESCAN_INTERVAL_MS) {
+ rescanAllControllers();
+ lastRescan = now;
+ }
+ try {
+ Thread.sleep(POLL_INTERVAL_MS);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ }
+ LOGGER.debug("NVMe namespace did not appear at " + address.getPath() +
" within " + waitSecs + "s");
+ return false;
+ }
+
+ private void rescanAllControllers() {
+ try {
+ File sysClass = new File("/sys/class/nvme");
+ File[] ctrls = sysClass.listFiles();
+ if (ctrls == null) {
+ return;
+ }
+ for (File ctrl : ctrls) {
+ Process p = new ProcessBuilder("nvme", "ns-rescan", "/dev/" +
ctrl.getName())
+ .redirectErrorStream(true).start();
+ if (!p.waitFor(NS_RESCAN_TIMEOUT_SECS, TimeUnit.SECONDS)) {
+ // Kill runaway nvme-cli invocations so they do not pile
+ // up under the JVM on every poll iteration while we
+ // are still waiting for the namespace to appear.
+ LOGGER.debug("nvme ns-rescan /dev/" + ctrl.getName()
+ + " did not complete within " +
NS_RESCAN_TIMEOUT_SECS
+ + "s; terminating");
+ p.destroyForcibly();
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.debug("nvme ns-rescan attempt failed: " + e.getMessage());
+ }
+ }
+
+ @Override
+ public boolean disconnectPhysicalDisk(String volumePath, KVMStoragePool
pool) {
+ // NVMe-oF: the kernel drops the namespace as soon as the target
removes
+ // this host's connection, so there is no host-side map to tear down
the way
+ // Fibre Channel must flush its device-mapper entry.
+ return true;
+ }
+
+ @Override
+ public boolean disconnectPhysicalDisk(Map<String, String>
volumeToDisconnect) {
+ return true;
+ }
+
+ @Override
+ public boolean disconnectPhysicalDiskByPath(String localPath) {
+ // Same rationale as disconnectPhysicalDisk above. Only claim paths
+ // that look like NVMe EUI symlinks so we don't swallow foreign paths.
+ return localPath != null &&
localPath.startsWith("/dev/disk/by-id/nvme-eui.");
+ }
+
+ @Override
+ public boolean deletePhysicalDisk(String uuid, KVMStoragePool pool,
Storage.ImageFormat format) {
+ // Namespaces are created and destroyed by the storage provider, never
from the
+ // host. Report "not handled here" instead of throwing, so a caller on
a cleanup
+ // path behaves the same as it does on the Fibre Channel adapter.
+ LOGGER.info("deletePhysicalDisk({}) not handled by the NVMe-oF
adapter; the storage provider owns namespace deletion", uuid);
+ return false;
+ }
+
+ @Override
+ public KVMPhysicalDisk createPhysicalDisk(String name, KVMStoragePool
pool, PhysicalDiskFormat format,
+ Storage.ProvisioningType provisioningType, long size, byte[]
passphrase) {
+ throw new UnsupportedOperationException("Unimplemented method
'createPhysicalDisk'");
+ }
+
+ @Override
+ public KVMPhysicalDisk createTemplateFromDisk(KVMPhysicalDisk disk, String
name, QemuImg.PhysicalDiskFormat format, long size, KVMStoragePool destPool) {
+ LOGGER.info("createTemplateFromDisk not supported on NVMe-oF pools");
+ return null;
+ }
+
+ @Override
+ public List<KVMPhysicalDisk> listPhysicalDisks(String storagePoolUuid,
KVMStoragePool pool) {
+ // The array owns the namespace inventory; it is not enumerable from
the host.
+ LOGGER.info("listPhysicalDisks not supported on NVMe-oF pool {}",
storagePoolUuid);
+ return null;
+ }
+
+ @Override
+ public KVMPhysicalDisk copyPhysicalDisk(KVMPhysicalDisk disk, String name,
KVMStoragePool destPool, int timeout) {
+ return copyPhysicalDisk(disk, name, destPool, timeout, null, null,
null);
+ }
+
+ /**
+ * Copy a template or source disk into a pre-provisioned NVMe namespace on
+ * this pool, so it can be consumed by a VM as a root or data volume.
+ *
+ * The destination namespace is expected to have already been created on
+ * the storage provider and connected to this host's hostgroup (that is
+ * the storage orchestrator's responsibility, not the KVM adapter's). All
+ * this method does is resolve the destination device path via
+ * {@link #getPhysicalDisk} - which will nvme ns-rescan and wait for the
+ * by-id/nvme-eui.<NGUID> symlink to show up if the kernel has not
+ * picked it up yet - and {@code qemu-img convert} the source image into
+ * the raw block device.
+ *
+ * User-space encryption passphrases are not supported: the provider
+ * already encrypts at rest and qemu-img LUKS on top of a shared
+ * hostgroup-scoped namespace is not a sensible layering.
+ */
+ @Override
+ public KVMPhysicalDisk copyPhysicalDisk(KVMPhysicalDisk disk, String name,
KVMStoragePool destPool, int timeout,
+ byte[] srcPassphrase, byte[] destPassphrase,
Storage.ProvisioningType provisioningType) {
+ if (disk == null || StringUtils.isEmpty(name) || destPool == null) {
+ throw new CloudRuntimeException("Unable to copy disk to NVMe-oF
pool: source disk, destination volume name or destination pool not specified");
+ }
+ if (srcPassphrase != null || destPassphrase != null) {
+ throw new CloudRuntimeException("NVMe-oF adapter does not support
user-space encrypted source or destination volumes");
+ }
+
+ KVMPhysicalDisk destDisk = destPool.getPhysicalDisk(name);
+ if (destDisk == null || StringUtils.isEmpty(destDisk.getPath())) {
+ throw new CloudRuntimeException("Unable to resolve NVMe namespace
for destination volume [" + name + "] on pool [" + destPool.getUuid() + "]");
+ }
+
+ destDisk.setFormat(QemuImg.PhysicalDiskFormat.RAW);
+ destDisk.setVirtualSize(disk.getVirtualSize());
+ destDisk.setSize(disk.getSize());
+
+ LOGGER.info(String.format("Copying source disk [path=%s, format=%s,
virtualSize=%d] to NVMe-oF namespace [path=%s] on pool [%s]",
+ disk.getPath(), disk.getFormat(), disk.getVirtualSize(),
destDisk.getPath(), destPool.getUuid()));
+
+ QemuImgFile srcFile = new QemuImgFile(disk.getPath(),
disk.getFormat());
+ QemuImgFile destFile = new QemuImgFile(destDisk.getPath(),
destDisk.getFormat());
+
+ try {
+ QemuImg qemu = new QemuImg(timeout);
+ qemu.convert(srcFile, destFile, true);
+ } catch (QemuImgException | LibvirtException e) {
+ throw new CloudRuntimeException("Failed to copy source disk [" +
disk.getPath() + "] to NVMe-oF namespace ["
+ + destDisk.getPath() + "] on pool [" + destPool.getUuid()
+ "]: " + e.getMessage(), e);
+ }
+
+ LOGGER.info("Successfully copied source disk to NVMe-oF namespace [" +
destDisk.getPath() + "] on pool [" + destPool.getUuid() + "]");
+ return destDisk;
+ }
+
+ @Override
+ public KVMPhysicalDisk createDiskFromTemplate(KVMPhysicalDisk template,
String name, PhysicalDiskFormat format, Storage.ProvisioningType
provisioningType, long size, KVMStoragePool destPool, int timeout, byte[]
passphrase) {
+ throw new UnsupportedOperationException("Unimplemented method
'createDiskFromTemplate'");
+ }
+
+ @Override
+ public KVMPhysicalDisk createDiskFromTemplateBacking(KVMPhysicalDisk
template, String name, PhysicalDiskFormat format, long size, KVMStoragePool
destPool, int timeout, byte[] passphrase) {
+ throw new UnsupportedOperationException("Unimplemented method
'createDiskFromTemplateBacking'");
+ }
+
+ /**
+ * Write a directly-downloaded template onto a namespace on this pool.
+ *
+ * The two path arguments are different kinds of thing, which is worth
being explicit
+ * about: {@code templateFilePath} is a plain local file produced by the
direct-download
+ * helper, while {@code destTemplatePath} is a managed volume path of the
form
+ * {@code type=NVMETCP;address=...}. Only the destination may be resolved
through
+ * {@link KVMStoragePool#getPhysicalDisk(String)}; passing the local file
through it
+ * would hand a file name to {@link #parseAndValidatePath(String)} and
fail. The caller
+ * has already issued connectPhysicalDisk() for the destination, so the
namespace is
+ * expected to be present.
+ *
+ * The template is written as QCOW2 onto the raw namespace rather than as
RAW. That
+ * matches the ScaleIO adaptor, is consistent with the QCOW2 format the
template is
+ * registered with, and is what lets the Qcow2Inspector check the caller
runs on the
+ * returned path succeed.
+ *
+ * Note what the caller does with the returned disk: KVMStorageProcessor
puts
+ * {@code disk.getName()} into the DirectDownloadAnswer, and that becomes
the template's
+ * install path and later its external name, which the provider
interpolates into array
+ * REST calls. {@link #getPhysicalDisk(String, KVMStoragePool)} names a
disk
+ * {@code AddressInfo.toString()}, which contains spaces and brackets and
would produce
+ * a name that cannot be placed in a URI. So the disk handed back here is
named with the
+ * managed volume path we were given, matching what a volume records.
+ */
+ @Override
+ public KVMPhysicalDisk createTemplateFromDirectDownloadFile(String
templateFilePath, String destTemplatePath, KVMStoragePool destPool,
Storage.ImageFormat format, int timeout) {
+ if (StringUtils.isAnyEmpty(templateFilePath, destTemplatePath) ||
destPool == null) {
+ throw new CloudRuntimeException("Unable to create a template from
a direct download file on an NVMe-oF pool: "
+ + "template file path, destination template path or
destination pool not specified");
+ }
+
+ if (!Storage.ImageFormat.QCOW2.equals(format) &&
!Storage.ImageFormat.RAW.equals(format)) {
+ throw new CloudRuntimeException("Unsupported direct download
template format for NVMe-oF pools: " + format
+ + "; expected " + Storage.ImageFormat.QCOW2 + " or " +
Storage.ImageFormat.RAW);
+ }
+
+ File sourceFile = new File(templateFilePath);
+ if (!sourceFile.exists()) {
+ throw new CloudRuntimeException("Direct download template file " +
templateFilePath + " does not exist on this host");
+ }
+
+ LOGGER.debug("Creating a template on NVMe-oF pool [{}] from direct
download file [{}] into [{}], format [{}]",
+ destPool.getUuid(), templateFilePath, destTemplatePath,
format);
+
+ String srcTemplateFilePath = templateFilePath;
+ KVMPhysicalDisk destDisk;
+ try {
+ destDisk = destPool.getPhysicalDisk(destTemplatePath);
+ if (destDisk == null || StringUtils.isEmpty(destDisk.getPath())) {
+ throw new CloudRuntimeException("Unable to resolve the NVMe
namespace for destination template path ["
+ + destTemplatePath + "] on pool [" +
destPool.getUuid() + "]");
+ }
+
+ // Direct-download templates are commonly published compressed.
+ if
(TemplateDownloaderUtil.isTemplateExtractable(templateFilePath)) {
+ srcTemplateFilePath = sourceFile.getParent() + "/" +
UUID.randomUUID().toString();
+ LOGGER.debug("Extracting downloaded template [{}] to [{}]",
templateFilePath, srcTemplateFilePath);
+
Script.runSimpleBashScript(TemplateDownloaderUtil.getExtractCommandForDownloadedFile(templateFilePath,
srcTemplateFilePath));
+ Script.runSimpleBashScript("rm -f " + templateFilePath);
+ }
+
+ QemuImg.PhysicalDiskFormat srcFormat =
Storage.ImageFormat.RAW.equals(format)
+ ? QemuImg.PhysicalDiskFormat.RAW :
QemuImg.PhysicalDiskFormat.QCOW2;
+
+ QemuImg qemu = new QemuImg(timeout);
+ QemuImgFile srcFile = new QemuImgFile(srcTemplateFilePath,
srcFormat);
+ // Populates the virtual size, and fails early if the file is
unreadable or is
+ // not in the format the template claims to be.
+ qemu.info(srcFile);
+
+ long namespaceSize = getPhysicalDiskSize(destDisk.getPath());
+ if (namespaceSize > 0 && srcFile.getSize() > namespaceSize) {
+ throw new CloudRuntimeException("Direct download template
needs " + srcFile.getSize()
+ + " bytes but the NVMe namespace at " +
destDisk.getPath() + " is only " + namespaceSize + " bytes");
+ }
+
+ QemuImgFile destFile = new QemuImgFile(destDisk.getPath(),
QemuImg.PhysicalDiskFormat.QCOW2);
+ destFile.setSize(srcFile.getSize());
+
+ LOGGER.debug("Converting [{}] onto NVMe namespace [{}]",
srcFile.getFileName(), destDisk.getPath());
+ qemu.create(destFile);
+ qemu.convert(srcFile, destFile);
+
+ KVMPhysicalDisk template = new KVMPhysicalDisk(destDisk.getPath(),
destTemplatePath, destPool);
+ template.setFormat(QemuImg.PhysicalDiskFormat.QCOW2);
+ template.setVirtualSize(srcFile.getSize());
+ template.setSize(srcFile.getSize());
+ destDisk = template;
+ LOGGER.info("Wrote direct download template onto NVMe namespace
[{}] on pool [{}]",
+ destDisk.getPath(), destPool.getUuid());
+ } catch (QemuImgException | LibvirtException e) {
+ throw new CloudRuntimeException("Failed to write the direct
download template [" + templateFilePath
+ + "] onto the NVMe namespace for [" + destTemplatePath +
"] on pool [" + destPool.getUuid()
+ + "]: " + e.getMessage(), e);
+ } finally {
+ // Only remove what we extracted; the original download belongs to
the caller.
+ if (!srcTemplateFilePath.equals(templateFilePath)) {
+ Script.runSimpleBashScript("rm -f " + srcTemplateFilePath);
+ }
+ }
+
+ return destDisk;
+ }
+
+ @Override
+ public boolean refresh(KVMStoragePool pool) {
+ return true;
+ }
+
+ @Override
+ public boolean createFolder(String uuid, String path) {
+ return createFolder(uuid, path, null);
+ }
+
+ @Override
+ public boolean createFolder(String uuid, String path, String localPath) {
+ // Block storage has no directory structure to create. Succeed rather
than
+ // throw, matching the Fibre Channel adapter.
+ LOGGER.info("createFolder({}, {}, {}) is a no-op on NVMe-oF pools",
uuid, path, localPath);
+ return true;
+ }
+
+ /**
+ * Host-side half of a volume resize. The storage provider has already
grown the
+ * namespace on the array by the time we get here, so all that remains is
to make
+ * the new capacity visible locally and tell a running guest about it.
+ *
+ * Unlike the SCSI/FC path there is no device-mapper map to grow: the
kernel picks
+ * up the new namespace size either from the target's asynchronous event
+ * notification or from an explicit {@code nvme ns-rescan}, which we issue
here
+ * rather than waiting for the AEN.
+ */
+ public void resize(String path, String vmName, long newSize) {
+ AddressInfo address = parseAndValidatePath(path);
+ if (address == null || address.getPath() == null) {
+ throw new CloudRuntimeException("Unable to resize NVMe-oF volume,
could not derive a device path from [" + path + "]");
+ }
+
+ LOGGER.debug("Resizing NVMe-oF volume " + address.getPath() + " to " +
newSize + " bytes for VM " + vmName);
+
+ rescanAllControllers();
+
+ long observed = waitForNamespaceSize(address.getPath(), newSize);
+ if (observed < newSize) {
+ // Not fatal: the array has already been grown, and the kernel may
still
+ // catch up via an AEN. Surface it rather than failing the
operation, so
+ // the management server does not roll back a resize that did
happen.
+ LOGGER.warn("NVMe namespace " + address.getPath() + " still
reports " + observed
+ + " bytes after rescan, expected at least " + newSize
+ + "; the guest may not observe the new size until the next
rescan");
+ }
+
+ notifyGuestOfResize(address.getPath(), vmName, newSize,
address.getAddress());
+ }
+
+ /**
+ * Poll the block device until it reports at least {@code expectedSize},
since
+ * ns-rescan and AEN processing are asynchronous.
+ *
+ * @return the last size observed, which may be smaller than expected on
timeout.
+ */
+ private long waitForNamespaceSize(String devicePath, long expectedSize) {
+ long deadline = System.currentTimeMillis() + RESIZE_SETTLE_TIMEOUT_MS;
+ long observed = getPhysicalDiskSize(devicePath);
+ while (observed < expectedSize && System.currentTimeMillis() <
deadline) {
+ try {
+ Thread.sleep(POLL_INTERVAL_MS);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ break;
+ }
+ rescanAllControllers();
+ observed = getPhysicalDiskSize(devicePath);
+ }
+ return observed;
+ }
+
+ /**
+ * Ask libvirt to re-read the size of the guest's block device, so a
running VM
+ * sees the extra capacity without a reboot. A stopped VM needs nothing
here: it
+ * picks up the new size when the disk is next attached.
+ */
+ private void notifyGuestOfResize(String devicePath, String vmName, long
newSize, String eui) {
+ if (StringUtils.isEmpty(vmName)) {
+ LOGGER.debug("No VM name supplied for resize of " + devicePath +
"; skipping guest notification");
+ return;
+ }
+
+ if (!isVmRunning(vmName)) {
+ LOGGER.debug("VM " + vmName + " is not running; skipping guest
notification for " + devicePath);
+ return;
+ }
+
+ String target = findDomainDiskTarget(vmName, devicePath, eui);
+ if (target == null) {
+ LOGGER.warn("Could not find a disk target for " + devicePath + "
in domain " + vmName
+ + "; skipping guest notification");
+ return;
+ }
+
+ // virsh blockresize takes the new size in KiB.
+ Script cmd = new Script("virsh", LOGGER);
+ cmd.add("blockresize");
+ cmd.add("--path", target);
+ cmd.add("--size", String.valueOf(newSize / 1024L));
+ cmd.add(vmName);
+ String result = cmd.execute();
+ if (result != null) {
+ LOGGER.warn("virsh blockresize of " + target + " on " + vmName + "
failed: " + result);
+ } else {
+ LOGGER.info("Notified " + vmName + " of new size " + newSize + "
bytes for " + target);
+ }
+ }
+
+ private boolean isVmRunning(String vmName) {
+ Script cmd = new Script("virsh", LOGGER);
+ cmd.add("domstate", vmName);
+ OutputInterpreter.OneLineParser parser = new
OutputInterpreter.OneLineParser();
+ String result = cmd.execute(parser);
+ return result == null && parser.getLine() != null &&
parser.getLine().trim().startsWith("running");
+ }
+
+ /**
+ * Resolve the domain-local disk target (vda, vdb, ...) backing {@code
devicePath}.
+ *
+ * libvirt reports the source as it was configured, but may instead
surface a
+ * canonicalised /dev/nvmeXnY in place of the /dev/disk/by-id symlink we
attached,
+ * so accept either form, falling back to matching the bare EUI.
+ */
+ private String findDomainDiskTarget(String vmName, String devicePath,
String eui) {
+ Script cmd = new Script("virsh", LOGGER);
+ cmd.add("domblklist", vmName);
+ OutputInterpreter.AllLinesParser parser = new
OutputInterpreter.AllLinesParser();
+ String result = cmd.execute(parser);
+ if (result != null || parser.getLines() == null) {
+ return null;
+ }
+ String canonical = resolveCanonicalPath(devicePath);
+ for (String line : parser.getLines().split("\\R")) {
+ String[] cols = line.trim().split("\\s+");
+ if (cols.length < 2) {
+ continue;
+ }
+ String source = cols[1];
+ if (source.equals(devicePath)
+ || (canonical != null && source.equals(canonical))
+ || (StringUtils.isNotEmpty(eui) &&
source.toLowerCase().contains(eui.toLowerCase()))) {
+ return cols[0];
+ }
+ }
+ return null;
+ }
+
+ /** Resolve a /dev/disk/by-id symlink to its /dev/nvmeXnY target, or null.
*/
+ private String resolveCanonicalPath(String devicePath) {
+ try {
+ return new File(devicePath).getCanonicalPath();
+ } catch (Exception e) {
+ LOGGER.debug("Could not canonicalise " + devicePath + ": " +
e.getMessage());
+ return null;
+ }
+ }
+
+ boolean isConnected(String path) {
+ Script test = new Script("/bin/test", LOGGER);
+ test.add("-b", path);
+ test.execute();
+ return test.getExitValue() == 0;
+ }
+
+ long getPhysicalDiskSize(String diskPath) {
+ if (StringUtils.isEmpty(diskPath)) {
+ return 0;
+ }
+ Script cmd = new Script("blockdev", LOGGER);
+ cmd.add("--getsize64", diskPath);
+ OutputInterpreter.OneLineParser parser = new
OutputInterpreter.OneLineParser();
+ String result = cmd.execute(parser);
+ if (result != null) {
+ LOGGER.debug("Unable to get the disk size at path: " + diskPath);
+ return 0;
+ }
+ try {
+ return Long.parseLong(parser.getLine());
+ } catch (NumberFormatException e) {
+ return 0;
+ }
+ }
+
+ private static String resolveHostnameShort() {
+ try {
+ String h = java.net.InetAddress.getLocalHost().getHostName();
+ int dot = h.indexOf('.');
+ return dot > 0 ? h.substring(0, dot) : h;
+ } catch (Exception e) {
+ return null;
+ }
+ }
+
+ private static String resolveHostnameFq() {
+ try {
+ return java.net.InetAddress.getLocalHost().getCanonicalHostName();
+ } catch (Exception e) {
+ return null;
+ }
+ }
+
+ /**
+ * Same shape as {@link MultipathSCSIAdapterBase.AddressInfo}. Kept
+ * separate so this class can be consumed by adapters that don't share the
+ * SCSI base.
+ */
+ public static final class AddressInfo {
+ String type;
+ String address;
+ String connectionId;
+ String path;
+
+ public AddressInfo(String type, String address, String connectionId,
String path) {
+ this.type = type;
+ this.address = address;
+ this.connectionId = connectionId;
+ this.path = path;
+ }
+
+ public String getType() { return type; }
+ public String getAddress() { return address; }
+ public String getConnectionId() { return connectionId; }
+ public String getPath() { return path; }
+
+ public String toString() {
+ return String.format("AddressInfo %s [address=%s, connectionId=%s,
path=%s]", type, address, connectionId, path);
+ }
+ }
+}
diff --git
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFPool.java
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFPool.java
new file mode 100644
index 00000000000..fec1e28f404
--- /dev/null
+++
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/MultipathNVMeOFPool.java
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package com.cloud.hypervisor.kvm.storage;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.cloudstack.utils.qemu.QemuImg;
+import org.apache.cloudstack.utils.qemu.QemuImg.PhysicalDiskFormat;
+import org.joda.time.Duration;
+
+import com.cloud.agent.api.to.HostTO;
+import com.cloud.hypervisor.kvm.resource.KVMHABase.HAStoragePool;
+import com.cloud.storage.Storage;
+import com.cloud.storage.Storage.ProvisioningType;
+
+/**
+ * KVMStoragePool for NVMe-over-Fabrics pools. Mirror of
+ * {@link MultipathSCSIPool} for adapters based on
+ * {@link MultipathNVMeOFAdapterBase}. Every data operation is delegated
+ * back to the adapter; the pool itself only tracks addressing/identity.
+ */
+public class MultipathNVMeOFPool implements KVMStoragePool {
+ private final String uuid;
+ private final String sourceHost;
+ private final int sourcePort;
+ private final String sourceDir;
+ private final Storage.StoragePoolType storagePoolType;
+ private final StorageAdaptor storageAdaptor;
+ private final Map<String, String> details;
+ private long capacity;
+ private long used;
+ private long available;
+
+ public MultipathNVMeOFPool(String uuid, String host, int port, String path,
+ Storage.StoragePoolType poolType, Map<String, String> poolDetails,
StorageAdaptor adaptor) {
+ this.uuid = uuid;
+ this.sourceHost = host;
+ this.sourcePort = port;
+ this.sourceDir = path;
+ this.storagePoolType = poolType;
+ this.storageAdaptor = adaptor;
+ this.details = poolDetails;
+ this.capacity = 0;
+ this.used = 0;
+ this.available = 0;
+ }
+
+ public MultipathNVMeOFPool(String uuid, StorageAdaptor adaptor) {
+ this.uuid = uuid;
+ this.sourceHost = null;
+ this.sourcePort = -1;
+ this.sourceDir = null;
+ this.storagePoolType = Storage.StoragePoolType.NVMeTCP;
+ this.storageAdaptor = adaptor;
+ this.details = new HashMap<>();
+ this.capacity = 0;
+ this.used = 0;
+ this.available = 0;
+ }
+
+ @Override
+ public KVMPhysicalDisk createPhysicalDisk(String volumeUuid,
ProvisioningType provisioningType, long size, byte[] passphrase) {
+ return null;
+ }
+
+ @Override
+ public KVMPhysicalDisk createPhysicalDisk(String volumeUuid,
PhysicalDiskFormat format, ProvisioningType provisioningType, long size, byte[]
passphrase) {
+ return null;
+ }
+
+ @Override
+ public boolean connectPhysicalDisk(String volumeUuid, Map<String, String>
details) {
+ return storageAdaptor.connectPhysicalDisk(volumeUuid, this, details,
false);
+ }
+
+ @Override
+ public KVMPhysicalDisk getPhysicalDisk(String volumeId) {
+ return storageAdaptor.getPhysicalDisk(volumeId, this);
+ }
+
+ @Override
+ public boolean disconnectPhysicalDisk(String volumeUuid) {
+ return storageAdaptor.disconnectPhysicalDisk(volumeUuid, this);
+ }
+
+ @Override
+ public boolean deletePhysicalDisk(String volumeUuid, Storage.ImageFormat
format) {
+ return true;
+ }
+
+ /**
+ * Host-side resize, mirroring {@link MultipathSCSIPool#resize}. The
provider has
+ * already grown the volume on the array; the adaptor makes it visible
here.
+ */
+ public void resize(String path, String vmName, long newSize) {
+ ((MultipathNVMeOFAdapterBase)storageAdaptor).resize(path, vmName,
newSize);
+ }
+
+ @Override
+ public List<KVMPhysicalDisk> listPhysicalDisks() {
+ return null;
+ }
+
+ @Override
+ public String getUuid() {
+ return uuid;
+ }
+
+ public void setCapacity(long capacity) { this.capacity = capacity; }
+ @Override public long getCapacity() { return this.capacity; }
+ public void setUsed(long used) { this.used = used; }
+ @Override public long getUsed() { return this.used; }
+ public void setAvailable(long available) { this.available = available; }
+ @Override public long getAvailable() { return this.available; }
+
+ @Override public boolean refresh() { return false; }
+ @Override public boolean isExternalSnapshot() { return true; }
+ @Override public String getLocalPath() { return null; }
+ @Override public String getSourceHost() { return this.sourceHost; }
+ @Override public String getSourceDir() { return this.sourceDir; }
+ @Override public int getSourcePort() { return this.sourcePort; }
+ @Override public String getAuthUserName() { return null; }
+ @Override public String getAuthSecret() { return null; }
+ @Override public Storage.StoragePoolType getType() { return
storagePoolType; }
+ @Override public boolean delete() { return false; }
+ @Override public QemuImg.PhysicalDiskFormat getDefaultFormat() { return
QemuImg.PhysicalDiskFormat.RAW; }
+ @Override public boolean createFolder(String path) { return false; }
+ @Override public boolean supportsConfigDriveIso() { return false; }
+ @Override public Map<String, String> getDetails() { return this.details; }
+ @Override public boolean isPoolSupportHA() { return false; }
+ @Override public String getHearthBeatPath() { return null; }
+
+ @Override
+ public String createHeartBeatCommand(HAStoragePool primaryStoragePool,
String hostPrivateIp, boolean hostValidation) {
+ return null;
+ }
+
+ @Override public String getStorageNodeId() { return null; }
+
+ @Override
+ public Boolean hasHeartBeat(HAStoragePool pool, HostTO host) { return
null; }
+
+ @Override
+ public Boolean hasVmActivity(HAStoragePool pool, HostTO host, Duration
activityScriptTimeout,
+ String volumeUUIDListString, String vmActivityCheckPath, long
duration) {
+ return null;
+ }
+}
diff --git
a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/NVMeTCPAdapter.java
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/NVMeTCPAdapter.java
new file mode 100644
index 00000000000..596f5d7bb9c
--- /dev/null
+++
b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/NVMeTCPAdapter.java
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package com.cloud.hypervisor.kvm.storage;
+
+import com.cloud.storage.Storage;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * StorageAdaptor for the {@link Storage.StoragePoolType#NVMeTCP} pool type.
+ * All operational logic lives in {@link MultipathNVMeOFAdapterBase}; this
+ * class just binds that logic to a pool type so
+ * {@link KVMStoragePoolManager} can find it via reflection.
+ */
+public class NVMeTCPAdapter extends MultipathNVMeOFAdapterBase {
+ private static final Logger LOGGER =
LogManager.getLogger(NVMeTCPAdapter.class);
+
+ public NVMeTCPAdapter() {
+ LOGGER.info("Loaded NVMeTCPAdapter for StorageLayer");
+ }
+
+ @Override
+ public String getName() {
+ return "NVMeTCPAdapter";
+ }
+
+ @Override
+ public Storage.StoragePoolType getStoragePoolType() {
+ return Storage.StoragePoolType.NVMeTCP;
+ }
+
+ @Override
+ public boolean isStoragePoolTypeSupported(Storage.StoragePoolType type) {
+ return Storage.StoragePoolType.NVMeTCP.equals(type);
+ }
+}
diff --git
a/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/adapter/ProviderVolume.java
b/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/adapter/ProviderVolume.java
index 25577903e3d..a954560508c 100644
---
a/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/adapter/ProviderVolume.java
+++
b/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/adapter/ProviderVolume.java
@@ -35,6 +35,7 @@ public interface ProviderVolume {
public String getExternalName();
public String getExternalConnectionId();
public enum AddressType {
- FIBERWWN
+ FIBERWWN,
+ NVMETCP
}
}
diff --git
a/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java
b/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java
index c8efc08c289..4714576498a 100644
---
a/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java
+++
b/plugins/storage/volume/adaptive/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/AdaptiveDataStoreLifeCycleImpl.java
@@ -179,7 +179,7 @@ public class AdaptiveDataStoreLifeCycleImpl extends
BasePrimaryDataStoreLifeCycl
parameters.setHost(uri.getHost());
parameters.setPort(uri.getPort());
parameters.setPath(uri.getPath() + "?" + uri.getQuery());
- parameters.setType(StoragePoolType.FiberChannel);
+ parameters.setType(pickPoolType(uri));
parameters.setZoneId(zoneId);
parameters.setPodId(podId);
parameters.setClusterId(clusterId);
@@ -402,4 +402,26 @@ public class AdaptiveDataStoreLifeCycleImpl extends
BasePrimaryDataStoreLifeCycl
logger.info("Disabling storage pool {}", store);
_dataStoreHelper.disable(store);
}
+
+ /**
+ * Resolve the CloudStack StoragePoolType from the provider URL. Adaptive
+ * plugins advertise the underlying fabric via a {@code transport=} query
+ * parameter on the URL; when absent we keep the legacy FiberChannel
+ * default for backwards compatibility with adapters that still assume it.
+ */
+ private static StoragePoolType pickPoolType(java.net.URL uri) {
+ String query = uri.getQuery();
+ if (query != null) {
+ for (String tok : query.split("&")) {
+ int i = tok.indexOf('=');
+ if (i > 0 && "transport".equalsIgnoreCase(tok.substring(0,
i))) {
+ String value = tok.substring(i + 1);
+ if ("nvme-tcp".equalsIgnoreCase(value)) {
+ return StoragePoolType.NVMeTCP;
+ }
+ }
+ }
+ }
+ return StoragePoolType.FiberChannel;
+ }
}
diff --git
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayAdapter.java
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayAdapter.java
index fae94bb1bea..4e12e3eb18d 100644
---
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayAdapter.java
+++
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayAdapter.java
@@ -30,6 +30,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.regex.Pattern;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
@@ -78,6 +79,9 @@ public class FlashArrayAdapter implements ProviderAdapter {
public static final String HOSTGROUP = "hostgroup";
public static final String STORAGE_POD = "pod";
+ public static final String TRANSPORT = "transport";
+ public static final String TRANSPORT_FC = "fc";
+ public static final String TRANSPORT_NVME_TCP = "nvme-tcp";
public static final String KEY_TTL = "keyttl";
public static final String CONNECT_TIMEOUT_MS = "connectTimeoutMs";
public static final String POST_COPY_WAIT_MS = "postCopyWaitMs";
@@ -90,6 +94,9 @@ public class FlashArrayAdapter implements ProviderAdapter {
private static final String API_LOGIN_VERSION_DEFAULT = "1.19";
private static final String API_VERSION_DEFAULT = "2.23";
+ /** A FlashArray NVMe namespace EUI-128 is exactly 32 hexadecimal
characters. */
+ private static final Pattern EUI128_PATTERN =
Pattern.compile("[0-9a-fA-F]{32}");
+
// URLs for which the legacy-auth deprecation WARN has already been
emitted,
// so we don't spam the logs once per refresh per pool while it's still
configured.
private static final Set<String> WARNED_LEGACY_URLS =
ConcurrentHashMap.newKeySet();
@@ -99,7 +106,7 @@ public class FlashArrayAdapter implements ProviderAdapter {
public String hostgroup = null;
private static final DateTimeFormatter DELETION_TIMESTAMP_FORMAT =
DateTimeFormatter.ofPattern("yyyyMMddHHmmss").withZone(ZoneOffset.UTC);
-
+ private AddressType volumeAddressType = AddressType.FIBERWWN;
private String username;
private String password;
private String accessToken;
@@ -133,13 +140,14 @@ public class FlashArrayAdapter implements ProviderAdapter
{
request, new TypeReference<FlashArrayList<FlashArrayVolume>>()
{
});
- return (ProviderVolume) getFlashArrayItem(list);
+ return withAddressType((FlashArrayVolume) getFlashArrayItem(list));
}
/**
- * Volumes must be added to a host set to be visable to the hosts.
- * the Hostset should contain all the hosts that are membrers of the zone
or
- * cluster (depending on Cloudstack Storage Pool configuration)
+ * Connect the volume to the host that was granted access. The storage
framework
+ * grants and revokes access one host at a time, so each host is given its
own
+ * connection: a host-group scoped connection is shared by every member
host and
+ * therefore cannot express the removal of a single host's access.
*/
@Override
public String attach(ProviderAdapterContext context,
ProviderAdapterDataObject dataObject, String hostname) {
@@ -160,14 +168,21 @@ public class FlashArrayAdapter implements ProviderAdapter
{
}
if (list == null || list.getItems() == null ||
list.getItems().size() == 0) {
- throw new RuntimeException("Volume attach did not return lun
information");
+ throw new RuntimeException("Volume attach did not return
connection information "
+ + "(expected lun for Fibre Channel or nsid for
NVMe-TCP)");
}
FlashArrayConnection connection = (FlashArrayConnection)
this.getFlashArrayItem(list);
+ if (AddressType.NVMETCP.equals(volumeAddressType)) {
+ // The FlashArray REST API does not return nsid in the
connections
+ // payload for NVMe-TCP. The namespace is identified on the
host by
+ // EUI-128 (see FlashArrayVolume.getAddress()); the value
returned
+ // here is stored by the driver only for informational
purposes.
+ return connection.getNsid() != null ? "" +
connection.getNsid() : "1";
+ }
if (connection.getLun() == null) {
throw new RuntimeException("Volume attach missing lun field");
}
-
return "" + connection.getLun();
} catch (Throwable e) {
@@ -179,15 +194,35 @@ public class FlashArrayAdapter implements ProviderAdapter
{
});
if (list != null && list.getItems() != null) {
for (FlashArrayConnection conn : list.getItems()) {
- if (conn.getHost() != null && conn.getHost().getName()
!= null &&
- (conn.getHost().getName().equals(hostname) ||
conn.getHost().getName().equals(hostname.substring(0, hostname.indexOf('.'))))
&&
- conn.getLun() != null) {
+ if (AddressType.NVMETCP.equals(volumeAddressType)) {
+ // Match the host-scoped connection this adapter
creates. A
+ // host-group scoped match is kept only as a
fallback, so that
+ // volumes still carrying a group connection
created by an
+ // earlier release continue to resolve here.
+ if (conn.getHost() != null &&
conn.getHost().getName() != null
+ &&
(conn.getHost().getName().equals(hostname)
+ || (hostname.indexOf('.') > 0
+ && conn.getHost().getName()
+ .equals(hostname.substring(0,
hostname.indexOf('.')))))) {
+ return conn.getNsid() != null ? "" +
conn.getNsid() : "1";
+ }
+ if (hostgroup != null && conn.getHostGroup() !=
null
+ && conn.getHostGroup().getName() != null
+ &&
conn.getHostGroup().getName().equals(hostgroup)) {
+ return conn.getNsid() != null ? "" +
conn.getNsid() : "1";
+ }
+ } else if (conn.getHost() != null &&
conn.getHost().getName() != null
+ && (conn.getHost().getName().equals(hostname)
+ || (hostname.indexOf('.') > 0
+ && conn.getHost().getName()
+ .equals(hostname.substring(0,
hostname.indexOf('.')))))
+ && conn.getLun() != null) {
return "" + conn.getLun();
}
}
- throw new RuntimeException("Volume lun is not found in
existing connection");
+ throw new RuntimeException("Volume connection identifier
(lun/nsid) not found in existing connection");
} else {
- throw new RuntimeException("Volume lun is not found in
existing connection");
+ throw new RuntimeException("Volume connection is not found
in existing connection list");
}
} else {
throw e;
@@ -198,12 +233,10 @@ public class FlashArrayAdapter implements ProviderAdapter
{
@Override
public void detach(ProviderAdapterContext context,
ProviderAdapterDataObject dataObject, String hostname) {
String volumeName = normalizeName(pod, dataObject.getExternalName());
- // hostname is always provided by cloudstack, but we will detach from
hostgroup
- // if this pool is configured to use hostgroup for attachments
- if (hostgroup != null) {
- DELETE("/connections?host_group_names=" + hostgroup +
"&volume_names=" + volumeName);
- }
-
+ // Only the connection for this host may be removed. A host-group
scoped
+ // connection is shared by every member host, so deleting it here
would revoke
+ // the volume from all of them -- including the host a live-migrating
VM has
+ // just moved to, which pulls the namespace out from under the running
guest.
FlashArrayHost host = getHost(hostname);
if (host != null) {
DELETE("/connections?host_names=" + host.getName() +
"&volume_names=" + volumeName);
@@ -307,14 +340,37 @@ public class FlashArrayAdapter implements ProviderAdapter
{
throw new RuntimeException("Invalid search criteria provided for
getVolumeByAddress");
}
- // only support WWN type addresses at this time.
- if (!ProviderVolume.AddressType.FIBERWWN.equals(addressType)) {
+ String serial;
+ if (ProviderVolume.AddressType.FIBERWWN.equals(addressType)) {
+ // Strip the NAA prefix (1 char) + Pure OUI to recover the volume
serial.
+ serial = address.substring(FlashArrayVolume.PURE_OUI.length() +
1).toUpperCase();
+ } else if (ProviderVolume.AddressType.NVMETCP.equals(addressType)) {
+ // Reverse the EUI-128 layout: serial = eui[2:16] + eui[22:32],
after
+ // stripping the optional "eui." prefix that appears in udev paths.
+ String eui = address.startsWith("eui.") ? address.substring(4) :
address;
+ if (eui == null || !EUI128_PATTERN.matcher(eui).matches()) {
+ throw new RuntimeException("Invalid NVMe-TCP EUI-128 address ["
+ + address + "]: expected 32 hexadecimal characters,
got "
+ + (eui == null ? "null" :
String.valueOf(eui.length())));
+ }
+ // Validate the FlashArray EUI-128 layout before deriving a serial
from it, so a
+ // malformed or tampered address cannot be mapped onto an
unintended volume:
+ // 00 + serial[0:14] + <Pure OUI> + serial[14:24]
+ if (!eui.startsWith("00")) {
+ throw new RuntimeException("Invalid NVMe-TCP EUI-128 address
[" + address
+ + "]: expected a \"00\" prefix for a FlashArray
namespace");
+ }
+ if (!eui.regionMatches(true, 16, FlashArrayVolume.PURE_OUI_EUI, 0,
+ FlashArrayVolume.PURE_OUI_EUI.length())) {
+ throw new RuntimeException("Invalid NVMe-TCP EUI-128 address
[" + address
+ + "]: expected the Pure Storage OUI [" +
FlashArrayVolume.PURE_OUI_EUI
+ + "] at offset 16");
+ }
+ serial = (eui.substring(2, 16) + eui.substring(22)).toUpperCase();
+ } else {
throw new RuntimeException(
"Invalid volume address type [" + addressType + "]
requested for volume search");
}
-
- // convert WWN to serial to search on. strip out WWN type # + Flash
OUI value
- String serial = address.substring(FlashArrayVolume.PURE_OUI.length() +
1).toUpperCase();
String query = "serial='" + serial + "'";
FlashArrayVolume volume = null;
@@ -328,7 +384,7 @@ public class FlashArrayAdapter implements ProviderAdapter {
return null;
}
- volume = (FlashArrayVolume) this.getFlashArrayItem(list);
+ volume = withAddressType((FlashArrayVolume)
this.getFlashArrayItem(list));
if (volume != null && volume.getAddress() == null) {
return null;
}
@@ -365,8 +421,11 @@ public class FlashArrayAdapter implements ProviderAdapter {
"/volume-snapshots?source_names=" +
sourceDataObject.getExternalName(), null,
new TypeReference<FlashArrayList<FlashArrayVolume>>() {
});
-
- return (FlashArrayVolume) getFlashArrayItem(list);
+ // Stamp the pool's volume address type so
ProviderSnapshot.getAddress()
+ // emits an NVMe EUI-128 on NVMe-TCP pools. Without this, the adaptive
+ // driver persists the snapshot with an FC-style WWN and subsequent
+ // revert/list operations cannot locate the namespace.
+ return withAddressType((FlashArrayVolume) getFlashArrayItem(list));
}
/**
@@ -409,7 +468,12 @@ public class FlashArrayAdapter implements ProviderAdapter {
"/volume-snapshots?names=" + dataObject.getExternalName(),
new TypeReference<FlashArrayList<FlashArrayVolume>>() {
});
- return (FlashArrayVolume) getFlashArrayItem(list);
+ // Stamp the pool's volume address type so
ProviderSnapshot.getAddress()
+ // emits an NVMe EUI-128 on NVMe-TCP pools instead of the FIBERWWN
+ // default. Without this, the adaptive driver persists the snapshot
+ // path with an FC-style WWN and revert/list fails to locate the
+ // namespace on the host.
+ return withAddressType((FlashArrayVolume) getFlashArrayItem(list));
}
@Override
@@ -437,10 +501,7 @@ public class FlashArrayAdapter implements ProviderAdapter {
if
(sourceDataObject.getType().equals(ProviderAdapterDataObject.Type.SNAPSHOT)) {
currentVol = getSnapshot(sourceDataObject.getExternalName());
} else {
- currentVol = (FlashArrayVolume) this
- .getFlashArrayItem(GET("/volumes?names=" +
sourceDataObject.getExternalName(),
- new
TypeReference<FlashArrayList<FlashArrayVolume>>() {
- }));
+ currentVol = getVolume(sourceDataObject.getExternalName());
}
if (currentVol == null) {
@@ -459,7 +520,7 @@ public class FlashArrayAdapter implements ProviderAdapter {
"/volumes?names=" + payload.getExternalName() +
"&overwrite=true", payload,
new TypeReference<FlashArrayList<FlashArrayVolume>>() {
});
- FlashArrayVolume outVolume = (FlashArrayVolume)
getFlashArrayItem(list);
+ FlashArrayVolume outVolume = withAddressType((FlashArrayVolume)
getFlashArrayItem(list));
pause(postCopyWait);
return outVolume;
}
@@ -764,6 +825,13 @@ public class FlashArrayAdapter implements ProviderAdapter {
}
}
+ String transport = connectionDetails.get(FlashArrayAdapter.TRANSPORT);
+ if (transport == null) {
+ transport = queryParms.get(FlashArrayAdapter.TRANSPORT);
+ }
+ volumeAddressType = TRANSPORT_NVME_TCP.equalsIgnoreCase(transport)
+ ? AddressType.NVMETCP : AddressType.FIBERWWN;
+
// retrieve for legacy purposes. if set, we'll remove any connections
to hostgroup we find and use the host
hostgroup = connectionDetails.get(FlashArrayAdapter.HOSTGROUP);
if (hostgroup == null) {
@@ -919,11 +987,17 @@ public class FlashArrayAdapter implements ProviderAdapter
{
}
}
+ /**
+ * Look up a volume by name. The pool's address type is always stamped
onto the
+ * result, so no caller can emit an FC-style WWN for a volume that lives
on an
+ * NVMe-TCP pool (see {@link FlashArrayVolume#getAddress()}). The snapshot
+ * accessor below holds the same invariant.
+ */
private FlashArrayVolume getVolume(String volumeName) {
FlashArrayList<FlashArrayVolume> list = GET("/volumes?names=" +
volumeName,
new TypeReference<FlashArrayList<FlashArrayVolume>>() {
});
- return (FlashArrayVolume) getFlashArrayItem(list);
+ return withAddressType((FlashArrayVolume) getFlashArrayItem(list));
}
private FlashArrayPod getVolumeNamespace(String name) {
@@ -937,7 +1011,14 @@ public class FlashArrayAdapter implements ProviderAdapter
{
FlashArrayList<FlashArrayVolume> list = GET("/volume-snapshots?names="
+ snapshotName,
new TypeReference<FlashArrayList<FlashArrayVolume>>() {
});
- return (FlashArrayVolume) getFlashArrayItem(list);
+ return withAddressType((FlashArrayVolume) getFlashArrayItem(list));
+ }
+
+ private FlashArrayVolume withAddressType(FlashArrayVolume vol) {
+ if (vol != null) {
+ vol.setAddressType(volumeAddressType);
+ }
+ return vol;
}
private Object getFlashArrayItem(FlashArrayList<?> list) {
@@ -1246,7 +1327,16 @@ public class FlashArrayAdapter implements
ProviderAdapter {
if (list != null && list.getItems() != null) {
for (FlashArrayConnection conn : list.getItems()) {
- if (conn.getHost() != null) {
+ if (AddressType.NVMETCP.equals(volumeAddressType)) {
+ // Key on the host name so connid.<hostname> is
matched by
+ // parseAndValidatePath. NVMe-TCP reports an nsid
where Fibre
+ // Channel reports a lun; the connection itself is
host-scoped
+ // on both transports.
+ if (conn.getHost() != null && conn.getHost().getName()
!= null) {
+ String id = conn.getNsid() != null ? "" +
conn.getNsid() : "1";
+ map.put(conn.getHost().getName(), id);
+ }
+ } else if (conn.getHost() != null) {
map.put(conn.getHost().getName(), "" + conn.getLun());
}
}
diff --git
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayConnection.java
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayConnection.java
index 76cec9f70c4..b115035fda2 100644
---
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayConnection.java
+++
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayConnection.java
@@ -31,6 +31,8 @@ public class FlashArrayConnection {
private FlashArrayVolume volume;
@JsonProperty("lun")
private Integer lun;
+ @JsonProperty("nsid")
+ private Integer nsid;
public FlashArrayConnectionHostgroup getHostGroup() {
return hostGroup;
@@ -64,5 +66,12 @@ public class FlashArrayConnection {
this.lun = lun;
}
+ public Integer getNsid() {
+ return nsid;
+ }
+
+ public void setNsid(Integer nsid) {
+ this.nsid = nsid;
+ }
}
diff --git
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayVolume.java
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayVolume.java
index a3201a753a7..8ae232cbc68 100644
---
a/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayVolume.java
+++
b/plugins/storage/volume/flasharray/src/main/java/org/apache/cloudstack/storage/datastore/adapter/flasharray/FlashArrayVolume.java
@@ -16,6 +16,8 @@
// under the License.
package org.apache.cloudstack.storage.datastore.adapter.flasharray;
+import java.util.regex.Pattern;
+
import org.apache.cloudstack.storage.datastore.adapter.ProviderSnapshot;
import com.fasterxml.jackson.annotation.JsonIgnore;
@@ -27,6 +29,13 @@ import com.fasterxml.jackson.annotation.JsonProperty;
@JsonInclude(JsonInclude.Include.NON_NULL)
public class FlashArrayVolume implements ProviderSnapshot {
public static final String PURE_OUI = "24a9370";
+ // The 3-byte OUI as it appears inside an NVMe EUI-128 (no trailing
nibble).
+ // FC WWNs use a 7-hex-digit Pure OUI; NVMe NGUIDs embed the same vendor
+ // prefix in its raw 6-hex-digit form.
+ public static final String PURE_OUI_EUI = "24a937";
+
+ /** FlashArray volume serials are exactly 24 hexadecimal characters. */
+ private static final Pattern SERIAL_PATTERN =
Pattern.compile("[0-9a-fA-F]{24}");
@JsonProperty("destroyed")
private Boolean destroyed;
@@ -107,6 +116,23 @@ public class FlashArrayVolume implements ProviderSnapshot {
@JsonIgnore
public String getAddress() {
if (serial == null) return null;
+ if (AddressType.NVMETCP.equals(addressType)) {
+ // EUI-128 layout for FlashArray NVMe namespaces:
+ // 00 + serial[0:14] + <Pure OUI (24a937)> + serial[14:24]
+ // This is the value the Linux kernel exposes as
+ // /dev/disk/by-id/nvme-eui.<result>
+ // Require an exact 24-hex-character serial. Accepting anything
longer and slicing
+ // the first 24 characters would silently map two distinct serials
sharing a prefix
+ // onto the same EUI, breaking volume identity; accepting anything
shorter cannot
+ // produce a valid 32-character EUI at all.
+ if (!SERIAL_PATTERN.matcher(serial).matches()) {
+ throw new RuntimeException("FlashArray serial [" + serial
+ + "] cannot be used to build an NVMe EUI-128 address "
+ + "(expected exactly 24 hexadecimal characters, got "
+ + serial.length() + ")");
+ }
+ return ("00" + serial.substring(0, 14) + PURE_OUI_EUI +
serial.substring(14, 24)).toLowerCase();
+ }
return ("6" + PURE_OUI + serial).toLowerCase();
}
@Override
diff --git a/server/src/main/java/com/cloud/api/ApiDBUtils.java
b/server/src/main/java/com/cloud/api/ApiDBUtils.java
index 44f29040513..bb9867fa622 100644
--- a/server/src/main/java/com/cloud/api/ApiDBUtils.java
+++ b/server/src/main/java/com/cloud/api/ApiDBUtils.java
@@ -1355,7 +1355,8 @@ public class ApiDBUtils {
StoragePoolType.PowerFlex,
StoragePoolType.CLVM,
StoragePoolType.Linstor,
-
StoragePoolType.FiberChannel).contains(pool.getPoolType())) {
+ StoragePoolType.FiberChannel,
+
StoragePoolType.NVMeTCP).contains(pool.getPoolType())) {
// This case will note the presence of non-qcow2 primary
stores, suggesting KVM without NFS. Otherwse,
// If this check is not passed, the hypervisor type will
remain OVM.
type = HypervisorType.KVM;
diff --git a/server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
b/server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
index 8674014addd..e81cc00564e 100644
--- a/server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
+++ b/server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java
@@ -2601,7 +2601,8 @@ public class VolumeApiServiceImpl extends ManagerBase
implements VolumeApiServic
if (storagePoolVO.isManaged() && !List.of(
Storage.StoragePoolType.PowerFlex,
-
Storage.StoragePoolType.FiberChannel).contains(storagePoolVO.getPoolType())) {
+ Storage.StoragePoolType.FiberChannel,
+
Storage.StoragePoolType.NVMeTCP).contains(storagePoolVO.getPoolType())) {
Long instanceId = volume.getInstanceId();
if (instanceId != null) {
diff --git a/ui/src/views/infra/AddPrimaryStorage.vue
b/ui/src/views/infra/AddPrimaryStorage.vue
index 7d189032f09..eeb3ebdd22c 100644
--- a/ui/src/views/infra/AddPrimaryStorage.vue
+++ b/ui/src/views/infra/AddPrimaryStorage.vue
@@ -242,7 +242,7 @@
</a-select>
</a-form-item>
</div>
- <div v-if="form.provider !== 'DefaultPrimary' && form.provider !==
'PowerFlex' && form.provider !== 'Linstor' && form.provider !== 'NetApp ONTAP'
&& form.protocol !== 'FiberChannel'">
+ <div v-if="form.provider !== 'DefaultPrimary' && form.provider !==
'PowerFlex' && form.provider !== 'Linstor' && form.provider !== 'NetApp ONTAP'
&& form.protocol !== 'FiberChannel' && form.protocol !== 'NVMeTCP'">
<a-form-item name="managed" ref="managed">
<template #label>
<tooltip-label :title="$t('label.ismanaged')"
:tooltip="apiParams.managed.description"/>
@@ -829,7 +829,10 @@ export default {
if (value === 'PowerFlex') {
this.protocols = ['custom']
this.form.protocol = 'custom'
- } else if (value === 'Flash Array' || value === 'Primera') {
+ } else if (value === 'Flash Array') {
+ this.protocols = ['FiberChannel', 'NVMeTCP']
+ this.form.protocol = 'FiberChannel'
+ } else if (value === 'Primera') {
this.protocols = ['FiberChannel']
this.form.protocol = 'FiberChannel'
} else if (value === 'NetApp ONTAP') {
@@ -960,6 +963,16 @@ export default {
params['details[0].api_username'] = values.flashArrayUsername
params['details[0].api_password'] = values.flashArrayPassword
url = values.flashArrayURL
+ if (values.protocol === 'NVMeTCP') {
+ // Set (or replace) the transport= query parameter rather than
blindly appending,
+ // so a URL already carrying transport= from a hand-crafted
operator entry is not
+ // left with two conflicting values.
+ if (/[?&]transport=/.test(url)) {
+ url = url.replace(/([?&])transport=[^&]*/,
'$1transport=nvme-tcp')
+ } else {
+ url = url + (url.indexOf('?') === -1 ? '?' : '&') +
'transport=nvme-tcp'
+ }
+ }
} else if (values.provider === 'NetApp ONTAP') {
params['details[0].storageIP'] = values.ontapIP
params['details[0].username'] = values.ontapUsername
