[
https://ovirt-jira.atlassian.net/browse/OVIRT-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=39986#comment-39986
]
Marcin Sobczyk commented on OVIRT-2837:
---------------------------------------
Hi,
I thought it's been removed already, but it seems it's not.
I'm the author of the currently used and newer implementation of
artifact collection in lago,
but from my experiments, I've learned that the extraction of wildcard
paths never worked.
Here's an email I wrote to Galit about it some time ago:
=================================
Hi Galit,
aaah yes - wildcard collection doesn't work - it never worked, even
before my changes.
TL; DR - we just need to remove wildcard stuff from "LagoInitFile.in"
files ("/tmp/otopi*", "/tmp/ovirt*").
If you're curious what really happens... :)
The old algorithm uses "SCPClient" from "scp" library to copy files.
"scp.get" function accepts, among others, two arguments - "remote_path"
and "local_path".
What we do, is we change slashes in "remote_path" to underscores and
pass it as "local_path":
https://github.com/lago-project/lago/blob/7024fb7cabbd4ebc87f3ad12e35e0f5832af7d56/lago/plugins/vm.py#L651
The effect is, we command "scp.get" to retrieve "/tmp/otopi*" and save
it as "_tmp_otopi*"... which of course makes no sense at all and doesn't
work...
The new implementation *could* work with wildcards because the
collection is divided into two stages:
https://github.com/lago-project/lago/blob/9803eeacd41b3f91cd6661a110aa0285aaf4b957/lago/plugins/vm.py#L313
First we do the "tar -> copy tar with ssh -> untar to tmpdir" thing and
*only then* we use "shutil.move" to rename the files to the underscored
version.
We could use "glob" module to try to iterate over stuff like
"/tmp/otopi*" and rename the files appropriately.
However, we maintain two parallel implementations of artifacts
collection - the old one being a "plan B" in case there's no "tar" or
"gzip" on the target machine.
This is the reason we have to keep both implementations identical in
behavior to avoid confusion. BTW the new implementation could drop the
underscore-renaming process completely - I think the only reason we do
the renaming in the old algorithm is because "scp" won't create
intermediary directories for you... untarring stuff handles that case
well, but that's a backwards-compatibility-breaking change :)
=================================
I will post a patch that removes this.
Regards, Marcin
On 11/18/19 2:45 PM, Amit Bawer wrote:
> Happens for several runs, full log can be seen at
> http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap.py/lago_logs/lago.log
> 2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR::
> - [Thread-42] lago-basic-suite-master-engine: [31mERROR [0m (in 0:00:08)
> 2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG::
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 398, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in
> extract_paths
> self._extract_paths_tar_gz(paths, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in
> wrapper
> return func(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in
> _extract_paths_tar_gz
> raise ExtractPathNoPathError(remote_path)
>
> 2019-11-18
> 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while
> running thread Thread-42
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 398, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 253, in
> extract_paths
> self._extract_paths_tar_gz(paths, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 102, in
> wrapper
> return func(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 341, in
> _extract_paths_tar_gz
> raise ExtractPathNoPathError(remote_path)
> ExtractPathNoPathError: Failed to extract files: /tmp/otopi*
>
> _______________________________________________
> Devel mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/[email protected]/message/PCS27XT3ODFL5ZRUPQMB5BFOW6BQXDGS/
> OST fails for collecting artifacts
> ----------------------------------
>
> Key: OVIRT-2837
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2837
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Amit Bawer
> Assignee: infra
>
> Happens for several runs, full log can be seen at
> http://jenkins.ovirt.org/job/ovirt-system-tests_manual/6057/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap.py/lago_logs/lago.log
> 2019-11-18 12:28:12,710::log_utils.py::end_log_task::670::root::ERROR::
> - [Thread-42] lago-basic-suite-master-engine: [31mERROR [0m (in
> 0:00:08)
> 2019-11-18 12:28:12,731::log_utils.py::__exit__::607::lago.prefix::DEBUG::
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 748, in collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 468, in extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
> line 398, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 253, in extract_paths
> self._extract_paths_tar_gz(paths, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 102, in wrapper
> return func(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 341, in _extract_paths_tar_gz
> raise ExtractPathNoPathError(remote_path)
> 2019-11-18
> 12:28:12,731::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error
> while running thread Thread-42
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526,
> in _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 748, in collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 468, in extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
> line 398, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 253, in extract_paths
> self._extract_paths_tar_gz(paths, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 102, in wrapper
> return func(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
> 341, in _extract_paths_tar_gz
> raise ExtractPathNoPathError(remote_path)
> ExtractPathNoPathError: Failed to extract files: /tmp/otopi*
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
_______________________________________________
Infra mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/LAJIK5VW46SPUEKPQ2T523VPFB4DNI7Q/
