Jeff is right, it's a limitation of kernel rbd, the recommendation is
to add `rbd default features = 3` to the configuration. I think there
are plans to support rbd-nbd in cinderlib which would allow using
additional features, but I'm not aware of anything concrete.
Additionally, the path for the cinderlib log is
/var/log/ovirt-engine/cinderlib/cinderlib.log, the error in this case
would appear in the vdsm.log on the relevant host, and would look
something like "RBD image feature set mismatch. You can disable
features unsupported by the kernel with 'rbd feature disable'"
On Wed, Oct 14, 2020 at 12:01 AM Michael Thomas <w...@caltech.edu> wrote:
>
> After getting past the proxy issue, I was finally able to run the
> engine-setup --reconfigure-optional-components. The new
> ManagedBlockStorage storage domain exists, and I was able to create a
> disk. However, I am unable to attach the disk to a running VM.
>
> The engine.log shows the following, with a reference to a possible
> cinderlib error ("cinderlib execution failed"):
>
> 2020-10-13 15:15:23,508-05 INFO
> [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-13)
> [c73386d0-a713-4c37-bc9b-e7c4f9083f78] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[grafana=VM_NAME]',
> sharedLocks='[5676d441-660e-4d9f-a586-e53ff0ea054b=VM]'}'
> 2020-10-13 15:15:23,522-05 INFO
> [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-13)
> [c73386d0-a713-4c37-bc9b-e7c4f9083f78] Running command: UpdateVmCommand
> internal: false. Entities affected : ID:
> 5676d441-660e-4d9f-a586-e53ff0ea054b Type: VMAction group
> EDIT_VM_PROPERTIES with role type USER
> 2020-10-13 15:15:23,536-05 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-13) [c73386d0-a713-4c37-bc9b-e7c4f9083f78] EVENT_ID:
> USER_UPDATE_VM(35), VM grafana configuration was updated by
> michael.thomas@internal-authz.
> 2020-10-13 15:15:23,539-05 INFO
> [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-13)
> [c73386d0-a713-4c37-bc9b-e7c4f9083f78] Lock freed to object
> 'EngineLock:{exclusiveLocks='[grafana=VM_NAME]',
> sharedLocks='[5676d441-660e-4d9f-a586-e53ff0ea054b=VM]'}'
> 2020-10-13 15:15:24,129-05 INFO
> [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default
> task-13) [f8829338-b040-46d0-a838-3cf28869637c] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[5419640e-445f-4b3f-a29d-b316ad031b7a=DISK]',
> sharedLocks=''}'
> 2020-10-13 15:15:24,147-05 INFO
> [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default
> task-13) [f8829338-b040-46d0-a838-3cf28869637c] Running command:
> AttachDiskToVmCommand internal: false. Entities affected : ID:
> 5676d441-660e-4d9f-a586-e53ff0ea054b Type: VMAction group
> CONFIGURE_VM_STORAGE with role type USER, ID:
> 5419640e-445f-4b3f-a29d-b316ad031b7a Type: DiskAction group ATTACH_DISK
> with role type USER
> 2020-10-13 15:15:24,152-05 INFO
> [org.ovirt.engine.core.bll.storage.disk.managedblock.ConnectManagedBlockStorageDeviceCommand]
> (default task-13) [7cb262cc] Running command:
> ConnectManagedBlockStorageDeviceCommand internal: true.
> 2020-10-13 15:15:26,006-05 ERROR
> [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor]
> (default task-13) [7cb262cc] cinderlib execution failed:
> 2020-10-13 15:15:26,011-05 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand]
> (default task-13) [7cb262cc] START, HotPlugDiskVDSCommand(HostName =
> ovirt4-mgmt.ldas.ligo-la.caltech.edu,
> HotPlugDiskVDSParameters:{hostId='61da4cdf-638b-4cbd-9921-5be820998d31',
> vmId='5676d441-660e-4d9f-a586-e53ff0ea054b',
> diskId='5419640e-445f-4b3f-a29d-b316ad031b7a'}), log id: 660ebc9e
> 2020-10-13 15:15:26,012-05 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand]
> (default task-13) [7cb262cc] Failed in 'HotPlugDiskVDS' method, for vds:
> 'ovirt4-mgmt.ldas.ligo-la.caltech.edu'; host:
> 'ovirt4-mgmt.ldas.ligo-la.caltech.edu': null
> 2020-10-13 15:15:26,012-05 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand]
> (default task-13) [7cb262cc] Command 'HotPlugDiskVDSCommand(HostName =
> ovirt4-mgmt.ldas.ligo-la.caltech.edu,
> HotPlugDiskVDSParameters:{hostId='61da4cdf-638b-4cbd-9921-5be820998d31',
> vmId='5676d441-660e-4d9f-a586-e53ff0ea054b',
> diskId='5419640e-445f-4b3f-a29d-b316ad031b7a'})' execution failed: null
> 2020-10-13 15:15:26,012-05 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand]
> (default task-13) [7cb262cc] FINISH, HotPlugDiskVDSCommand, return: ,
> log id: 660ebc9e
> 2020-10-13 15:15:26,012-05 ERROR
> [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default
> task-13) [7cb262cc] Command
> 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand' failed:
> EngineException: java.lang.NullPointerException (Failed with error
> ENGINE and code 5001)
> 2020-10-13 15:15:26,013-05 ERROR
> [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default
> task-13) [7cb262cc] Transaction rolled-back for command
> 'org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand'.
> 2020-10-13 15:15:26,021-05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-13) [7cb262cc] EVENT_ID:
> USER_FAILED_ATTACH_DISK_TO_VM(2,017), Failed to attach Disk testvm_disk
> to VM grafana (User: michael.thomas@internal-authz).
> 2020-10-13 15:15:26,021-05 INFO
> [org.ovirt.engine.core.bll.storage.disk.AttachDiskToVmCommand] (default
> task-13) [7cb262cc] Lock freed to object
> 'EngineLock:{exclusiveLocks='[5419640e-445f-4b3f-a29d-b316ad031b7a=DISK]',
> sharedLocks=''}'
>
> The /var/log/cinder/ directory on the ovirt node is empty, and doesn't
> exist on the engine itself.
>
> To verify that it's not a cephx permission issue, I tried accessing the
> block storage from both the engine and the ovirt node using the
> credentials I set up in the ManagedBlockStorage setup page:
>
> [root@ovirt4]# rbd --id ovirt ls rbd.ovirt.data
> volume-5419640e-445f-4b3f-a29d-b316ad031b7a
> [root@ovirt4]# rbd --id ovirt info
> rbd.ovirt.data/volume-5419640e-445f-4b3f-a29d-b316ad031b7a
> rbd image 'volume-5419640e-445f-4b3f-a29d-b316ad031b7a':
> size 100 GiB in 25600 objects
> order 22 (4 MiB objects)
> snapshot_count: 0
> id: 68a7cd6aeb3924
> block_name_prefix: rbd_data.68a7cd6aeb3924
> format: 2
> features: layering, exclusive-lock, object-map, fast-diff,
> deep-flatten
> op_features:
> flags:
> create_timestamp: Tue Oct 13 06:53:55 2020
> access_timestamp: Tue Oct 13 06:53:55 2020
> modify_timestamp: Tue Oct 13 06:53:55 2020
>
> Where else can I look to see where it's failing?
>
> --Mike
>
> On 9/30/20 2:19 AM, Benny Zlotnik wrote:
> > When you ran `engine-setup` did you enable cinderlib preview (it will
> > not be enabled by default)?
> > It should handle the creation of the database automatically, if you
> > didn't you can enable it by running:
> > `engine-setup --reconfigure-optional-components`
> >
> >
> > On Wed, Sep 30, 2020 at 1:58 AM Michael Thomas <w...@caltech.edu> wrote:
> >>
> >> Hi Benny,
> >>
> >> Thanks for the confirmation. I've installed openstack-ussuri and ceph
> >> Octopus. Then I tried using these instructions, as well as the deep
> >> dive that Eyal has posted at https://www.youtube.com/watch?v=F3JttBkjsX8.
> >>
> >> I've done this a couple of times, and each time the engine fails when I
> >> try to add the new managed block storage domain. The error on the
> >> screen indicates that it can't connect to the cinder database. The
> >> error in the engine log is:
> >>
> >> 2020-09-29 17:02:11,859-05 WARN
> >> [org.ovirt.engine.core.bll.storage.domain.AddManagedBlockStorageDomainCommand]
> >> (default task-2) [d519088c-7956-4078-b5cf-156e5b3f1e59] Validation of
> >> action 'AddManagedBlockStorageDomain' failed for user
> >> admin@internal-authz. Reasons:
> >> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_CINDERLIB_DATA_BASE_REQUIRED,ACTION_TYPE_FAILED_CINDERLIB_DATA_BASE_REQUIRED
> >>
> >> I had created the db on the engine with this command:
> >>
> >> su - postgres -c "psql -d template1 -c \"create database cinder owner
> >> engine template template0 encoding 'UTF8' lc_collate 'en_US.UTF-8'
> >> lc_ctype 'en_US.UTF-8';\""
> >>
> >> ...and added the following to the end of /var/lib/pgsql/data/pg_hba.conf:
> >>
> >> host cinder engine ::0/0 md5
> >> host cinder engine 0.0.0.0/0 md5
> >>
> >> Is there anywhere else I should look to find out what may have gone wrong?
> >>
> >> --Mike
> >>
> >> On 9/29/20 3:34 PM, Benny Zlotnik wrote:
> >>> The feature is currently in tech preview, but it's being worked on.
> >>> The feature page is outdated, but I believe this is what most users
> >>> in the mailing list were using. We held off on updating it because the
> >>> installation instructions have been a moving target, but it is more
> >>> stable now and I will update it soon.
> >>>
> >>> Specifically speaking, the openstack version should be updated to
> >>> train (it is likely ussuri works fine too, but I haven't tried it) and
> >>> cinderlib has an RPM now (python3-cinderlib)[1], so it can be
> >>> installed instead of using pip, same goes for os-brick. The rest of
> >>> the information is valid.
> >>>
> >>>
> >>> [1]
> >>> http://mirror.centos.org/centos/8/cloud/x86_64/openstack-ussuri/Packages/p/
> >>>
> >>> On Tue, Sep 29, 2020 at 10:37 PM Michael Thomas <w...@caltech.edu> wrote:
> >>>>
> >>>> I'm looking for the latest documentation for setting up a Managed Block
> >>>> Device storage domain so that I can move some of my VM images to ceph
> >>>> rbd.
> >>>>
> >>>> I found this:
> >>>>
> >>>> https://ovirt.org/develop/release-management/features/storage/cinderlib-integration.html
> >>>>
> >>>> ...but it has a big note at the top that it is "...not user
> >>>> documentation and should not be treated as such."
> >>>>
> >>>> The oVirt administration guide[1] does not talk about managed block
> >>>> devices.
> >>>>
> >>>> I've found a few mailing list threads that discuss people setting up a
> >>>> Managed Block Device with ceph, but didn't see any links to
> >>>> documentation steps that folks were following.
> >>>>
> >>>> Is the Managed Block Storage domain a supported feature in oVirt 4.4.2,
> >>>> and if so, where is the documentation for using it?
> >>>>
> >>>> --Mike
> >>>> [1]ovirt.org/documentation/administration_guide/
> >>>> _______________________________________________
> >>>> Users mailing list -- users@ovirt.org
> >>>> To unsubscribe send an email to users-le...@ovirt.org
> >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> >>>> oVirt Code of Conduct:
> >>>> https://www.ovirt.org/community/about/community-guidelines/
> >>>> List Archives:
> >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHCLXVOCELHOR3G7SH3GDPGRKITCW7UY/
> >>>
> >>
> >
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AANNLBPCRY46DOXRV24KRCEYW7GSEUJC/
