[ovirt-users] Re: Certificates expired...
Is a change in /etc/pki/vdsm/cert/cacert.pem on the nodes going to disrupt the communications between nodes and the engine? The procedure I followed blew away all of /etc/pki/vdsm on each node. I saved the old one. Jason On 8/4/23 14:38, Jason P. Thomas wrote: I restarted vdsmd and libvirtd after the cert update on each host. Jason On 8/4/23 14:34, Derek Atkins wrote: Did you restart vdsm after updating the certs? -derek On Fri, August 4, 2023 2:12 pm, Jason P. Thomas wrote: I updated the VDSM certs on the hosts and the apache cert on the engine. I'm guessing something is wrong with however the engine interacts with vdsm, I just don't know exactly what to do about it. Jason On 8/4/23 14:00, Derek Atkins wrote: Sounds like the Host Certs need to be updated.. Or possibly even the Engine CA Cert. -derek On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote: Konstantin, Right after I sent the email I got the engine running. The libvirt-spice certs had incorrect ownership. It still is not connecting to anything. Error in Events on the Engine is now: "VDSM command Get Host Capabilities failed: General SSLEngine problem" So status right now is, all VMs are running. Engine web ui is accessible. Engine shows all hosts as unassigned or Connecting or NonResponsive with repeated entries of the above error in Events. Sincerely, Jason On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote: Now the engine won't start at all and I'm afraid I'm one power outage away from complete disaster. I need to keep the old location up and functioning for another 4-6 months, so any insights would be greatly appreciated. Hi, 'engine won't start at all' can mean two things: 1) OS can't boot and thus you can't do SSH. Assuming that we are talking self-hosted engine, then you need to use command like below on host that runs ovengine VM (virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list and hosted-engine --vm-status might be helpful, VM should at least start to boot in order for you to achieve connectivity via console): hosted-engine --add-console-password --password=somepassword and then connect via VNC to IP that you will see in output and password that you used 2) ovirt-engine service can't start In that case it is likely that you will find reason of that in journalctl -u ovirt-engine --no-pager (/var/log/ovirt-engine/engine.log) BR, Konstantin ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/L3HNNMVKBOSHVMZFROSF4JW7PG36GBUQ/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SJM4VP7PYURO77GLIY2POBGBNTE3WMNH/
[ovirt-users] Re: How restore nodes ovirt UP from NonResponsive and VMs executing
Hello, The node (ovirt2) however is having consistent problems. The follow sequence of events is reproducible and is causing the host to enter a "NonOperational" state on the cluster: * Host ovirt2 installed * VDSM ovirt2 command ConnectStorageServerVDS failed: Message timeout which can be caused by communication issues * Host ovirt2 is not responding. Host cannot be fenced automatically because power management for the host is disabled. * Host ovirt2 cannot access the Storage Domain(s) attached to the Data Center DataCenter1. Setting Host state to Non-Operational. (5/27/1912:43:22 PM) * (Banner appears in GUI) Failed Activating Host ovirt2.witsconsult.com * Failed to connect Host ovirt2 to Storage Pool DataCenter1 (5/27/1912:47:07 PM) * Host ovirt2 cannot access the Storage Domain(s) attached to the Data Center DataCenter1. Setting Host state to Non-Operational. (5/27/1912:47:07 PM) * Host ovirt2 is not responding. Host cannot be fenced automatically because power management for the host is disabled. (5/27/1912:47:07 PM) * VDSM ovirt2 command ConnectStorageServerVDS failed: Message timeout which can be caused by communication issues (5/27/1912:47:07 PM) I can then re-activate ovirt2, which appears as green for approximately 5 minutes and then repeats all of the above issues. What can I do to troubleshoot this? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/CNCXB5HRYXL4CRP7NRUURQ3ILDAHKNTN/
[ovirt-users] ovirt node NonResponsive
Hello, I have ovirt with two nodes and one that are NonResponsive and and cant manage them because they are in Unknown state. It seems that nodes lost connection for a while with their gateway. The node (ovirt2) however is having consistent problems. The follow sequence of events is reproducible and is causing the host to enter a "NonOperational" state on the cluster: What is the proper way of restoring management? I have a two-node cluster with the ovirt manager running standlone on the virtual maachine CentOS-Stream-9 and the ovirt node running the most recent oVirt Node 4.5.4 software. I can then re-activate ovirt2, which appears as green for approximately 5 minutes and then repeats all of the above issues. What can I do to troubleshoot this? ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R25AFAZWTAJA4QLKXMVBELWMBUQSG7FA/
[ovirt-users] Re: Certificates expired...
cen, apache.p12 was the first snowflake in this avalanche. I did find something showing how to generate a new one and install it. That actually allowed me to access the engine web interface again. Kinda useless since the engine can't talk to any of the nodes though. Haha. Thanks for the info. I'll look into the engine.p12 between sessions of updating my resume. Haha Thanks, Jason On 8/8/23 17:30, cen wrote: Hi, I went through a similar ordeal half a year ago and forgot all the exact procedures already but for me, in the end after following all the guides and replacing the "standard" certs it was either engine.p12 or apache.p12 keystore that also had outdated certs (apparently mTLS is being used!). Updating these keystores is not documented anywhere. No idea if you are in the same situation but wanted to throw this out there. Best regards, cen On 4. 08. 23 20:12, Jason P. Thomas wrote: I updated the VDSM certs on the hosts and the apache cert on the engine. I'm guessing something is wrong with however the engine interacts with vdsm, I just don't know exactly what to do about it. Jason On 8/4/23 14:00, Derek Atkins wrote: Sounds like the Host Certs need to be updated.. Or possibly even the Engine CA Cert. -derek On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote: Konstantin, Right after I sent the email I got the engine running. The libvirt-spice certs had incorrect ownership. It still is not connecting to anything. Error in Events on the Engine is now: "VDSM command Get Host Capabilities failed: General SSLEngine problem" So status right now is, all VMs are running. Engine web ui is accessible. Engine shows all hosts as unassigned or Connecting or NonResponsive with repeated entries of the above error in Events. Sincerely, Jason On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote: Now the engine won't start at all and I'm afraid I'm one power outage away from complete disaster. I need to keep the old location up and functioning for another 4-6 months, so any insights would be greatly appreciated. Hi, 'engine won't start at all' can mean two things: 1) OS can't boot and thus you can't do SSH. Assuming that we are talking self-hosted engine, then you need to use command like below on host that runs ovengine VM (virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list and hosted-engine --vm-status might be helpful, VM should at least start to boot in order for you to achieve connectivity via console): hosted-engine --add-console-password --password=somepassword and then connect via VNC to IP that you will see in output and password that you used 2) ovirt-engine service can't start In that case it is likely that you will find reason of that in journalctl -u ovirt-engine --no-pager (/var/log/ovirt-engine/engine.log) BR, Konstantin ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AMVZEWY45QHPEDHJQZGJMZWESN2RZBPB/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UQZ7DE6UWAINN3UJ7OMBW7G2WQ25W2YX/
[ovirt-users] Re: oVirt 4.5.5 snapshot - Migration failed due to an Error: Fatal error during migration
Hi Jorge,
To get reliable migration, I had to have a 2 Gbps link. Before that, I
would migrate one VM at a time to avoid migration failures.
On Thu, Aug 10, 2023 at 5:45 PM Jorge Visentini
wrote:
> Any tips about this error?
>
> 2023-08-10 18:24:57,544-03 INFO
> [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-4)
> [633be3a0-3afd-490c-b412-805d2b14e1c2] Lock Acquired to object
> 'EngineLock:{exclusiveLocks='[29032e83-cfaf-4d30-bcc2-df72c5358552=VM]',
> sharedLocks=''}'
> 2023-08-10 18:24:57,578-03 INFO
> [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-4)
> [633be3a0-3afd-490c-b412-805d2b14e1c2] Running command:
> MigrateVmToServerCommand internal: false. Entities affected : ID:
> 29032e83-cfaf-4d30-bcc2-df72c5358552 Type: VMAction group MIGRATE_VM with
> role type USER
> 2023-08-10 18:24:57,628-03 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-4)
> [633be3a0-3afd-490c-b412-805d2b14e1c2] START, MigrateVDSCommand(
> MigrateVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7',
> vmId='29032e83-cfaf-4d30-bcc2-df72c5358552', srcHost='ksmmi1r01ovirt18',
> dstVdsId='73c38b36-36da-4ffa-b17a-492fd7b093ae',
> dstHost='ksmmi1r01ovirt19:54321', migrationMethod='ONLINE',
> tunnelMigration='false', migrationDowntime='0', autoConverge='true',
> migrateCompressed='false', migrateEncrypted='false', consoleAddress='null',
> maxBandwidth='3125', parallel='null', enableGuestEvents='true',
> maxIncomingMigrations='2', maxOutgoingMigrations='2',
> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],
> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
> action={name=setDowntime, params=[200]}}, {limit=3,
> action={name=setDowntime, params=[300]}}, {limit=4,
> action={name=setDowntime, params=[400]}}, {limit=6,
> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,
> params=[]}}]]', dstQemu='10.250.156.19', cpusets='null',
> numaNodesets='null'}), log id: 5bbc21d6
> 2023-08-10 18:24:57,628-03 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] START,
> MigrateBrokerVDSCommand(HostName = ksmmi1r01ovirt18,
> MigrateVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7',
> vmId='29032e83-cfaf-4d30-bcc2-df72c5358552', srcHost='ksmmi1r01ovirt18',
> dstVdsId='73c38b36-36da-4ffa-b17a-492fd7b093ae',
> dstHost='ksmmi1r01ovirt19:54321', migrationMethod='ONLINE',
> tunnelMigration='false', migrationDowntime='0', autoConverge='true',
> migrateCompressed='false', migrateEncrypted='false', consoleAddress='null',
> maxBandwidth='3125', parallel='null', enableGuestEvents='true',
> maxIncomingMigrations='2', maxOutgoingMigrations='2',
> convergenceSchedule='[init=[{name=setDowntime, params=[100]}],
> stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
> action={name=setDowntime, params=[200]}}, {limit=3,
> action={name=setDowntime, params=[300]}}, {limit=4,
> action={name=setDowntime, params=[400]}}, {limit=6,
> action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort,
> params=[]}}]]', dstQemu='10.250.156.19', cpusets='null',
> numaNodesets='null'}), log id: 14d92c9
> 2023-08-10 18:24:57,631-03 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] FINISH,
> MigrateBrokerVDSCommand, return: , log id: 14d92c9
> 2023-08-10 18:24:57,634-03 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-4)
> [633be3a0-3afd-490c-b412-805d2b14e1c2] FINISH, MigrateVDSCommand, return:
> MigratingFrom, log id: 5bbc21d6
> 2023-08-10 18:24:57,639-03 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-4) [633be3a0-3afd-490c-b412-805d2b14e1c2] EVENT_ID:
> VM_MIGRATION_START(62), Migration started (VM: ROUTER, Source:
> ksmmi1r01ovirt18, Destination: ksmmi1r01ovirt19, User: admin@ovirt
> @internalkeycloak-authz).
> 2023-08-10 18:24:57,641-03 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-13) [] VM
> '29032e83-cfaf-4d30-bcc2-df72c5358552'(ROUTER) moved from 'MigratingFrom'
> --> 'Up'
> 2023-08-10 18:24:57,641-03 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-13) [] Adding VM
> '29032e83-cfaf-4d30-bcc2-df72c5358552'(ROUTER) to re-run list
> 2023-08-10 18:24:57,643-03 ERROR
> [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (ForkJoinPool-1-worker-13) [] Rerun VM
> '29032e83-cfaf-4d30-bcc2-df72c5358552'. Called from VDS 'ksmmi1r01ovirt18'
> 2023-08-10 18:24:57,679-03 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (EE-ManagedThreadFactory-engine-Thread-2194) [] START,
> MigrateStatusVDSCommand(HostName = ksmmi1r01ovirt18,
> MigrateStatusVDSCommandParameters:{hostId='282b69aa-8b74-4312-8cc0-9c20e01982b7',
>
[ovirt-users] Re: Certificates expired...
cen, apache.p12 was the first snowflake in this avalanche. I did find something showing how to generate a new one and install it. That actually allowed me to access the engine web interface again. Kinda useless since the engine can't talk to any of the nodes though. Haha. Thanks for the info. I'll look into the engine.p12 between sessions of updating my resume. Haha Thanks, Jason On 8/8/23 17:30, cen wrote: Hi, I went through a similar ordeal half a year ago and forgot all the exact procedures already but for me, in the end after following all the guides and replacing the "standard" certs it was either engine.p12 or apache.p12 keystore that also had outdated certs (apparently mTLS is being used!). Updating these keystores is not documented anywhere. No idea if you are in the same situation but wanted to throw this out there. Best regards, cen On 4. 08. 23 20:12, Jason P. Thomas wrote: I updated the VDSM certs on the hosts and the apache cert on the engine. I'm guessing something is wrong with however the engine interacts with vdsm, I just don't know exactly what to do about it. Jason On 8/4/23 14:00, Derek Atkins wrote: Sounds like the Host Certs need to be updated.. Or possibly even the Engine CA Cert. -derek On Fri, August 4, 2023 1:45 pm, Jason P. Thomas wrote: Konstantin, Right after I sent the email I got the engine running. The libvirt-spice certs had incorrect ownership. It still is not connecting to anything. Error in Events on the Engine is now: "VDSM command Get Host Capabilities failed: General SSLEngine problem" So status right now is, all VMs are running. Engine web ui is accessible. Engine shows all hosts as unassigned or Connecting or NonResponsive with repeated entries of the above error in Events. Sincerely, Jason On 8/4/23 13:08, konstantin.volenbovskyi--- via Users wrote: Now the engine won't start at all and I'm afraid I'm one power outage away from complete disaster. I need to keep the old location up and functioning for another 4-6 months, so any insights would be greatly appreciated. Hi, 'engine won't start at all' can mean two things: 1) OS can't boot and thus you can't do SSH. Assuming that we are talking self-hosted engine, then you need to use command like below on host that runs ovengine VM (virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf list and hosted-engine --vm-status might be helpful, VM should at least start to boot in order for you to achieve connectivity via console): hosted-engine --add-console-password --password=somepassword and then connect via VNC to IP that you will see in output and password that you used 2) ovirt-engine service can't start In that case it is likely that you will find reason of that in journalctl -u ovirt-engine --no-pager (/var/log/ovirt-engine/engine.log) BR, Konstantin ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PL4Q64G6IFUUW5TYVJWSMMIMXHBT3SSD/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H3M4O4TN67NZZPVXGPTO6CEBFEM47LET/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3GFW2SRSZB5QHNY3ABXG2KPQ6ZA36M5I/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AMVZEWY45QHPEDHJQZGJMZWESN2RZBPB/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WFDTM3I275O4UJKBT6PHFCZAKOUQDIYG/
