the restart was not on the SPM so it should not have caused this task to not be cleaned, i think that the task was just still running.
On Thu, Jul 5, 2018 at 10:44 AM, Tal Nisan <[email protected]> wrote: > The task is stuck and never cleared after the restart or we just have to > wait for it to be finished and cleared? > > On Thu, Jul 5, 2018 at 12:30 PM, Dafna Ron <[email protected]> wrote: > >> Jira opened: https://ovirt-jira.atlassian.net/browse/OVIRT-2286 >> >> Tal, can you help fix the test? >> >> thanks, >> Dafna >> >> >> On Wed, Jul 4, 2018 at 7:41 PM, Dafna Ron <[email protected]> wrote: >> >>> The vdsm restart is a test in basic_sanity.py >>> The task that is stuck is downloadImageFromStream >>> >>> 2018-07-04 10:12:52,659-04 WARN [org.ovirt.engine.core.bll.sto >>> rage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (default >>> task-1) [ce1c28ba-1550-457f-b5e3-ad051488f897] There are running tasks: >>> 'AsyncTask:{commandId='0b18c13f-0fce- >>> 4303-8f85-ae5a2b051991', >>> rootCommandId='0b18c13f-0fce-4303-8f85-ae5a2b051991', >>> storagePoolId='8dd2fe5a-9dca-42e2-8593-1de1b18b4887', >>> taskId='f2af86fb-dbbb-430c-afd9-2f25131583b1', >>> vdsmTaskId='45ee0fc8-830d-47d5-9c4a-6d4ed72ae6a1', stepId= >>> 'null', taskType='downloadImageFromStream', status='running'} >>> >>> The task succeeded but after the deactivate storage domain attempt: >>> >>> 2018-07-04 10:13:01,579-04 INFO [org.ovirt.engine.core.bll.Ser >>> ialChildCommandsExecutionCallback] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-98) >>> [1c611686] Command 'ProcessOvfUpdateForStorageDomain' id: >>> '6342fce4-96ff-4ae3-8b40-8155a >>> 5509761' child commands '[0b18c13f-0fce-4303-8f85-ae5a2b051991, >>> 8e16826f-93dd-442a-8a74-13d14222d45e]' executions were completed, >>> status 'SUCCEEDED' >>> >>> >>> we should have a solution in ost for locked objects failing jobs... will >>> open a Jira to follow >>> >>> >>> >>> On Wed, Jul 4, 2018 at 6:16 PM, Nir Soffer <[email protected]> wrote: >>> >>>> On Wed, Jul 4, 2018 at 6:46 PM Dafna Ron <[email protected]> wrote: >>>> >>>>> The actual test has failed with error: >>>>> >>>>> 2018-07-04 10:12:52,665-04 ERROR [org.ovirt.engine.api.restapi. >>>>> resource.AbstractBackendResource] (default task-1) [] Operation >>>>> Failed: [Cannot deactivate Storage while there are running tasks on this >>>>> Storage. >>>>> >>>>> -Please wait until tasks will finish and try again.] >>>>> >>>>> However, there is a problem with vdsm on host-1. it restarts which may >>>>> cause the issue with the running tasks. >>>>> >>>> >>>> Who restarted vdsm? why? >>>> >>>> >>>>> 2018-07-04 10:08:46,603-0400 INFO (ioprocess/5191) [IOProcessClient] >>>>> shutdown requested (__init__:108) >>>>> 2018-07-04 10:08:46,604-0400 INFO (MainThread) [storage.udev] >>>>> Stopping multipath event listener (udev:149) >>>>> 2018-07-04 10:08:46,604-0400 INFO (MainThread) [vdsm.api] FINISH >>>>> prepareForShutdown return=None from=internal, >>>>> task_id=bf3b87e4-febf-4cb6-8bfa-5840fc926b49 >>>>> (api:52) >>>>> 2018-07-04 10:08:46,605-0400 INFO (MainThread) [vds] Stopping threads >>>>> (vdsmd:160) >>>>> 2018-07-04 10:08:46,605-0400 INFO (MainThread) [vds] Exiting >>>>> (vdsmd:171) >>>>> 2018-07-04 10:10:00,145-0400 INFO (MainThread) [vds] (PID: 14034) I >>>>> am the actual vdsm 4.20.32-1.el7 lago-basic-suite-4-2-host-1 >>>>> (3.10.0-862.2.3.el7.x86_64) (vdsmd:149) >>>>> 2018-07-04 10:10:00,146-0400 INFO (MainThread) [vds] VDSM will run >>>>> with cpu affinity: frozenset([1]) (vdsmd:262) >>>>> 2018-07-04 10:10:00,151-0400 INFO (MainThread) [storage.HSM] START >>>>> HSM init (hsm:366) >>>>> 2018-07-04 10:10:00,154-0400 INFO (MainThread) [storage.HSM] Creating >>>>> data-center mount directory '/rhev/data-center/mnt' (hsm:373) >>>>> 2018-07-04 10:10:00,154-0400 INFO (MainThread) [storage.fileUtils] >>>>> Creating directory: /rhev/data-center/mnt mode: None (fileUtils:197) >>>>> 2018-07-04 10:10:00,265-0400 INFO (MainThread) [storage.HSM] >>>>> Unlinking file '/rhev/data-center/8dd2fe5a-9d >>>>> ca-42e2-8593-1de1b18b4887/44eba8db-3a9c-4fbe-ba33-a039fcd561e1' >>>>> (hsm:523) >>>>> 2018-07-04 10:10:00,266-0400 INFO (MainThread) [storage.HSM] >>>>> Unlinking file '/rhev/data-center/8dd2fe5a-9d >>>>> ca-42e2-8593-1de1b18b4887/mastersd' (hsm:523) >>>>> 2018-07-04 10:10:00,266-0400 INFO (MainThread) [storage.HSM] >>>>> Unlinking file '/rhev/data-center/8dd2fe5a-9d >>>>> ca-42e2-8593-1de1b18b4887/c7980a1e-91ef-4095-82eb-37ec03da9b3f' >>>>> (hsm:523) >>>>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.HSM] >>>>> Unlinking file '/rhev/data-center/8dd2fe5a-9d >>>>> ca-42e2-8593-1de1b18b4887/4fc62763-d8a5-4c36-8687-91870a92ff05' >>>>> (hsm:523) >>>>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.HSM] >>>>> Unlinking file '/rhev/data-center/8dd2fe5a-9d >>>>> ca-42e2-8593-1de1b18b4887/02363608-01b9-4176-b7a1-e9ee235f792a' >>>>> (hsm:523) >>>>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.udev] >>>>> Registering multipath event monitor <vdsm.storage.mpathhealth.Monitor >>>>> object at 0x7f9aa4548150> (udev:182) >>>>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.udev] >>>>> Starting multipath event listener (udev:116) >>>>> 2018-07-04 10:10:00,298-0400 INFO (MainThread) [storage.check] >>>>> Starting check service (check:91) >>>>> 2018-07-04 10:10:00,303-0400 INFO (MainThread) [storage.Dispatcher] >>>>> Starting StorageDispatcher... (di >>>>> >>>>> On Wed, Jul 4, 2018 at 4:12 PM, Greg Sheremeta <[email protected]> >>>>> wrote: >>>>> >>>>>> """ >>>>>> Error: Fault reason is "Operation Failed". Fault detail is "[Cannot >>>>>> deactivate Storage while there are running tasks on this Storage. >>>>>> -Please wait until tasks will finish and try again.]". HTTP response >>>>>> code is 409. >>>>>> """ >>>>>> >>>>>> >>>>>> On Wed, Jul 4, 2018 at 10:56 AM Ehud Yonasi <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Link to job: >>>>>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2571/ >>>>>>> >>>>>>> Suspected error: >>>>>>> >>>>>>> Error occured, aborting >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, >>>>>>> in do_run >>>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line >>>>>>> 184, in do_run >>>>>>> self._do_run(**vars(args)) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 505, in >>>>>>> wrapper >>>>>>> return func(*args, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 516, in >>>>>>> wrapper >>>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in >>>>>>> do_ovirt_runtest >>>>>>> raise RuntimeError('Some tests failed') >>>>>>> RuntimeError: Some tests failed >>>>>>> + res=1 >>>>>>> + cd - >>>>>>> /home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests >>>>>>> + return 1 >>>>>>> + failed=true >>>>>>> + env_collect >>>>>>> /home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/test_logs/basic-suite-4.2/post-007_sd_reattach.py >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Infra mailing list -- [email protected] >>>>>>> To unsubscribe send an email to [email protected] >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>>>> y/about/community-guidelines/ >>>>>>> List Archives: https://lists.ovirt.org/archiv >>>>>>> es/list/[email protected]/message/JVGNSGB3HDB3E7Y7CTQR2FCJ3KEXMLFW/ >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> GREG SHEREMETA >>>>>> >>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>> >>>>>> Red Hat NA >>>>>> >>>>>> <https://www.redhat.com/> >>>>>> >>>>>> [email protected] IRC: gshereme >>>>>> <https://red.ht/sig> >>>>>> >>>>>> _______________________________________________ >>>>>> Infra mailing list -- [email protected] >>>>>> To unsubscribe send an email to [email protected] >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>>> y/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archiv >>>>>> es/list/[email protected]/message/GRWM4DDAGDV52QM3AWMKLWJP7MNI7S5O/ >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- [email protected] >>>>> To unsubscribe send an email to [email protected] >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>> y/about/community-guidelines/ >>>>> List Archives: https://lists.ovirt.org/archiv >>>>> es/list/[email protected]/message/EPWWZ3WCO43XEDB53XPJYZL4PGTKOYDQ/ >>>>> >>>> >>> >> >
_______________________________________________ Infra mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/G4ZPZP7WGB7SGLAGF3Z2RGYKSANQFMM2/
