Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Mon, Nov 9, 2020 at 11:03 AM Richard W.M. Jones wrote:
>
> On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote:
> > On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David wrote:
> > >
> > > On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones
> > > wrote:
> > > >
> > > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
> > > > > wrote:
> > > > > >
> > > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > > > I think this is the right solution - when virt-something tool
> > > > > > > fails,
> > > > > > > it should log the reason for the failure - the error that caused
> > > > > > > the
> > > > > > > tool to fail. I'm not sure this is easy to do as the failing code
> > > > > > > run inside a special VM. Maybe the code running in the VM should
> > > > > > > log
> > > > > > > the output in a machine readable way, so once an error is detected
> > > > > > > virt-something can report the error as the reason, without running
> > > > > > > in debug mode.
> > > > > >
> > > > > > All the virt-* tools that I've written have a non-zero exit code and
> > > > > > print an error message on stderr when they fail. Errors from inside
> > > > > > the appliance are propagated to the library and thence to the tool
> > > > > > correctly.
> > > > > >
> > > > > > I think the best thing to do is:
> > > > > >
> > > > > > - spool up stdout + stderr from the tool
> > > > > >
> > > > > > - if the exit code != 0, save the spooled output for analysis
> > > > > >
> > > > > > - if the exit code == 0, discard it (or keep it if you like)
> > > > >
> > > > > This is what we already do, and the result is not helpful. If you look
> > > > > at the log message in the previous message, basically the only
> > > > > info about the error is:
> > > > >
> > > > > libguestfs error: guestfs_launch failed
> > > > >
> > > > > I don't see what we can do with this error message.
> > > >
> > > > Right, so in this particular instance the error message would tell us
> > > > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > > > is broken in some way :-/
> > > >
> > > > There's not a particularly good answer here if you don't want to ever
> > > > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > > > libguestfs-test-tool if you see any error which matches the substring
> > > > /guestfs_launch/ ?
> > >
> > > Another (orthogonal?) option:
> > >
> > > Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> > > (e.g. some other file descriptor, or to a file passed via env or
> > > whatever).
> > > This way, it might make sense for vdsm to always pass these vars, continue
> > > logging all stdout/err, and log/keep debug/trace logs only on errors.
> >
> > This now happened again:
> >
> > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/
> >
> > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
> >
> > 2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
> > getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
> > 'Done'}, 'io_tune_policies_dict':
> > {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> > 'current_values': [{'name': 'vda', 'path':
> > '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> > 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> > 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
> > 'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
> > 2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
> > Return 'Host.getAllVmIoTunePolicies' in bridge with
> > {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> > 'current_values': [{'name': 'vda', 'path':
> > '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> > 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> > 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
> > 'read_iops_sec': 0}}]}} (__init__:360)
> > 2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
> > = b"virt-sparsify: error: libguestfs error: guestfs_launch
> > failed.\nThis usually means the libguestfs appliance failed to start
> > or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
> > run the command again. For further information, read:\n
> > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
> > also run 'libguestfs-test-tool' and post the *complete* output\ninto a
> > bug report or message to the libguestfs mailing list.\n\nIf reporting
> > bugs, run virt-sparsify with debugging enabled and
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote:
> On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David wrote:
> >
> > On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones
> > wrote:
> > >
> > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
> > > > wrote:
> > > > >
> > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > > I think this is the right solution - when virt-something tool fails,
> > > > > > it should log the reason for the failure - the error that caused the
> > > > > > tool to fail. I'm not sure this is easy to do as the failing code
> > > > > > run inside a special VM. Maybe the code running in the VM should log
> > > > > > the output in a machine readable way, so once an error is detected
> > > > > > virt-something can report the error as the reason, without running
> > > > > > in debug mode.
> > > > >
> > > > > All the virt-* tools that I've written have a non-zero exit code and
> > > > > print an error message on stderr when they fail. Errors from inside
> > > > > the appliance are propagated to the library and thence to the tool
> > > > > correctly.
> > > > >
> > > > > I think the best thing to do is:
> > > > >
> > > > > - spool up stdout + stderr from the tool
> > > > >
> > > > > - if the exit code != 0, save the spooled output for analysis
> > > > >
> > > > > - if the exit code == 0, discard it (or keep it if you like)
> > > >
> > > > This is what we already do, and the result is not helpful. If you look
> > > > at the log message in the previous message, basically the only
> > > > info about the error is:
> > > >
> > > > libguestfs error: guestfs_launch failed
> > > >
> > > > I don't see what we can do with this error message.
> > >
> > > Right, so in this particular instance the error message would tell us
> > > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > > is broken in some way :-/
> > >
> > > There's not a particularly good answer here if you don't want to ever
> > > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > > libguestfs-test-tool if you see any error which matches the substring
> > > /guestfs_launch/ ?
> >
> > Another (orthogonal?) option:
> >
> > Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> > (e.g. some other file descriptor, or to a file passed via env or whatever).
> > This way, it might make sense for vdsm to always pass these vars, continue
> > logging all stdout/err, and log/keep debug/trace logs only on errors.
>
> This now happened again:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
>
> 2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
> getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
> 'Done'}, 'io_tune_policies_dict':
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
> '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
> 'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
> 2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
> Return 'Host.getAllVmIoTunePolicies' in bridge with
> {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
> 'current_values': [{'name': 'vda', 'path':
> '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
> 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
> 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
> 'read_iops_sec': 0}}]}} (__init__:360)
> 2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
> = b"virt-sparsify: error: libguestfs error: guestfs_launch
> failed.\nThis usually means the libguestfs appliance failed to start
> or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
> run the command again. For further information, read:\n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
> also run 'libguestfs-test-tool' and post the *complete* output\ninto a
> bug report or message to the libguestfs mailing list.\n\nIf reporting
> bugs, run virt-sparsify with debugging enabled and include the
> \ncomplete output:\n\n virt-sparsify -v -x [...]\n"; = 1
> (commands:98)
>
> I suggest that if we have come to a dead-end and no-one has any clue, then
> we either patch something (vdsm?) to allow getting more information if this
> happens again, or open a bug for
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David wrote:
>
> On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones wrote:
> >
> > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
> > > wrote:
> > > >
> > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > > I think this is the right solution - when virt-something tool fails,
> > > > > it should log the reason for the failure - the error that caused the
> > > > > tool to fail. I'm not sure this is easy to do as the failing code
> > > > > run inside a special VM. Maybe the code running in the VM should log
> > > > > the output in a machine readable way, so once an error is detected
> > > > > virt-something can report the error as the reason, without running
> > > > > in debug mode.
> > > >
> > > > All the virt-* tools that I've written have a non-zero exit code and
> > > > print an error message on stderr when they fail. Errors from inside
> > > > the appliance are propagated to the library and thence to the tool
> > > > correctly.
> > > >
> > > > I think the best thing to do is:
> > > >
> > > > - spool up stdout + stderr from the tool
> > > >
> > > > - if the exit code != 0, save the spooled output for analysis
> > > >
> > > > - if the exit code == 0, discard it (or keep it if you like)
> > >
> > > This is what we already do, and the result is not helpful. If you look
> > > at the log message in the previous message, basically the only
> > > info about the error is:
> > >
> > > libguestfs error: guestfs_launch failed
> > >
> > > I don't see what we can do with this error message.
> >
> > Right, so in this particular instance the error message would tell us
> > that you should run libguestfs-test-tool because your qemu/kernel/etc
> > is broken in some way :-/
> >
> > There's not a particularly good answer here if you don't want to ever
> > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> > libguestfs-test-tool if you see any error which matches the substring
> > /guestfs_launch/ ?
>
> Another (orthogonal?) option:
>
> Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
> (e.g. some other file descriptor, or to a file passed via env or whatever).
> This way, it might make sense for vdsm to always pass these vars, continue
> logging all stdout/err, and log/keep debug/trace logs only on errors.
This now happened again:
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/
https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'code': 0, 'message':
'Done'}, 'io_tune_policies_dict':
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,34002 (api:54)
2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer]
Return 'Host.getAllVmIoTunePolicies' in bridge with
{'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [],
'current_values': [{'name': 'vda', 'path':
'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}} (__init__:360)
2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED:
= b"virt-sparsify: error: libguestfs error: guestfs_launch
failed.\nThis usually means the libguestfs appliance failed to start
or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand
run the command again. For further information, read:\n
http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can
also run 'libguestfs-test-tool' and post the *complete* output\ninto a
bug report or message to the libguestfs mailing list.\n\nIf reporting
bugs, run virt-sparsify with debugging enabled and include the
\ncomplete output:\n\n virt-sparsify -v -x [...]\n"; = 1
(commands:98)
I suggest that if we have come to a dead-end and no-one has any clue, then
we either patch something (vdsm?) to allow getting more information if this
happens again, or open a bug for further discussion/prioritization.
Best regards,
--
Didi
___
Libguestfs mailing list
Libguestfs@redhat.com
https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones wrote: > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote: > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones > > wrote: > > > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote: > > > > I think this is the right solution - when virt-something tool fails, > > > > it should log the reason for the failure - the error that caused the > > > > tool to fail. I'm not sure this is easy to do as the failing code > > > > run inside a special VM. Maybe the code running in the VM should log > > > > the output in a machine readable way, so once an error is detected > > > > virt-something can report the error as the reason, without running > > > > in debug mode. > > > > > > All the virt-* tools that I've written have a non-zero exit code and > > > print an error message on stderr when they fail. Errors from inside > > > the appliance are propagated to the library and thence to the tool > > > correctly. > > > > > > I think the best thing to do is: > > > > > > - spool up stdout + stderr from the tool > > > > > > - if the exit code != 0, save the spooled output for analysis > > > > > > - if the exit code == 0, discard it (or keep it if you like) > > > > This is what we already do, and the result is not helpful. If you look > > at the log message in the previous message, basically the only > > info about the error is: > > > > libguestfs error: guestfs_launch failed > > > > I don't see what we can do with this error message. > > Right, so in this particular instance the error message would tell us > that you should run libguestfs-test-tool because your qemu/kernel/etc > is broken in some way :-/ > > There's not a particularly good answer here if you don't want to ever > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run > libguestfs-test-tool if you see any error which matches the substring > /guestfs_launch/ ? Another (orthogonal?) option: Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err (e.g. some other file descriptor, or to a file passed via env or whatever). This way, it might make sense for vdsm to always pass these vars, continue logging all stdout/err, and log/keep debug/trace logs only on errors. Best regards, -- Didi ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote: > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones wrote: > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote: > > > I think this is the right solution - when virt-something tool fails, > > > it should log the reason for the failure - the error that caused the > > > tool to fail. I'm not sure this is easy to do as the failing code > > > run inside a special VM. Maybe the code running in the VM should log > > > the output in a machine readable way, so once an error is detected > > > virt-something can report the error as the reason, without running > > > in debug mode. > > > > All the virt-* tools that I've written have a non-zero exit code and > > print an error message on stderr when they fail. Errors from inside > > the appliance are propagated to the library and thence to the tool > > correctly. > > > > I think the best thing to do is: > > > > - spool up stdout + stderr from the tool > > > > - if the exit code != 0, save the spooled output for analysis > > > > - if the exit code == 0, discard it (or keep it if you like) > > This is what we already do, and the result is not helpful. If you look > at the log message in the previous message, basically the only > info about the error is: > > libguestfs error: guestfs_launch failed > > I don't see what we can do with this error message. Right, so in this particular instance the error message would tell us that you should run libguestfs-test-tool because your qemu/kernel/etc is broken in some way :-/ There's not a particularly good answer here if you don't want to ever use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run libguestfs-test-tool if you see any error which matches the substring /guestfs_launch/ ? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones wrote: > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote: > > I think this is the right solution - when virt-something tool fails, > > it should log the reason for the failure - the error that caused the > > tool to fail. I'm not sure this is easy to do as the failing code > > run inside a special VM. Maybe the code running in the VM should log > > the output in a machine readable way, so once an error is detected > > virt-something can report the error as the reason, without running > > in debug mode. > > All the virt-* tools that I've written have a non-zero exit code and > print an error message on stderr when they fail. Errors from inside > the appliance are propagated to the library and thence to the tool > correctly. > > I think the best thing to do is: > > - spool up stdout + stderr from the tool > > - if the exit code != 0, save the spooled output for analysis > > - if the exit code == 0, discard it (or keep it if you like) This is what we already do, and the result is not helpful. If you look at the log message in the previous message, basically the only info about the error is: libguestfs error: guestfs_launch failed I don't see what we can do with this error message. If this is an error running qemu, the error message from qemu can help to understand why qemu failed to start. > Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be > quite minimal for all tools. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-df lists disk usage of guests without needing to install any > software inside the virtual machine. Supports Linux and Windows. > http://people.redhat.com/~rjones/virt-df/ > ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote: > I think this is the right solution - when virt-something tool fails, > it should log the reason for the failure - the error that caused the > tool to fail. I'm not sure this is easy to do as the failing code > run inside a special VM. Maybe the code running in the VM should log > the output in a machine readable way, so once an error is detected > virt-something can report the error as the reason, without running > in debug mode. All the virt-* tools that I've written have a non-zero exit code and print an error message on stderr when they fail. Errors from inside the appliance are propagated to the library and thence to the tool correctly. I think the best thing to do is: - spool up stdout + stderr from the tool - if the exit code != 0, save the spooled output for analysis - if the exit code == 0, discard it (or keep it if you like) Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be quite minimal for all tools. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ ___ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)
On Mon, Oct 12, 2020 at 9:05 AM Yedidyah Bar David wrote:
>
> Hi all,
>
> On Mon, Oct 12, 2020 at 5:17 AM wrote:
> >
> > Project:
> > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
> > Build:
> > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/479/
>
> Above failed with:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/479/testReport/junit/basic-suite-master.test-scenarios/004_basic_sanity/test_sparsify_disk1/
>
> vdsm.log has:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/479/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log
>
> 2020-10-11 22:05:14,695-0400 INFO (jsonrpc/1) [api.host] FINISH
> getJobs return={'jobs': {'05eaea44-7e4c-4442-9926-2bcb696520f1':
> {'id': '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed',
> 'description': 'sparsify_volume', 'job_type': 'storage', 'error':
> {'code': 100, 'message': 'General Exception: (\'Command
> [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\',
> \\\'--in-place\\\',
> \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\']
> failed with rc=1 out=b\\\'3/12n{ "message": "libguestfs error:
> guestfs_launch failed.nThis usually means the libguestfs
> appliance failed to start or crashed.nDo:n export
> LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1nand run the command
> again. For further information, read:n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfsnYou
> can also run \\\'libguestfs-test-tool\\\' and post the
> *complete* outputninto a bug report or message to the
> libguestfs mailing list.", "timestamp":
> "2020-10-11T22:05:08.397538670-04:00", "type": "error" }n\\\'
> err=b"virt-sparsify: error: libguestfs error: guestfs_launch
> failed.nThis usually means the libguestfs appliance failed to
> start or crashed.nDo:n export LIBGUESTFS_DEBUG=1
> LIBGUESTFS_TRACE=1nand run the command again. For further
> information, read:n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfsnYou
> can also run \\\'libguestfs-test-tool\\\' and post the *complete*
> outputninto a bug report or message to the libguestfs mailing
> list.nnIf reporting bugs, run virt-sparsify with debugging
> enabled and include the ncomplete output:nn virt-sparsify
> -v -x [...]n"\',)'}}}, 'status': {'code': 0, 'message': 'Done'}}
> from=:::192.168.201.4,43318,
> flow_id=365642f4-2fe2-45df-937a-f4ca435eea38 (api:54)
> 2020-10-11 22:05:14,695-0400 DEBUG (jsonrpc/1) [jsonrpc.JsonRpcServer]
> Return 'Host.getJobs' in bridge with
> {'05eaea44-7e4c-4442-9926-2bcb696520f1': {'id':
> '05eaea44-7e4c-4442-9926-2bcb696520f1', 'status': 'failed',
> 'description': 'sparsify_volume', 'job_type': 'storage', 'error':
> {'code': 100, 'message': 'General Exception: (\'Command
> [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\',
> \\\'--in-place\\\',
> \\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/8b292c13-fd8a-4a7c-903c-5724ec742c10/images/a367c179-2ac9-4930-abeb-848229f81c97/515fcf06-8743-45d1-9af8-61a0c48e8c67\\\']
> failed with rc=1 out=b\\\'3/12n{ "message": "libguestfs error:
> guestfs_launch failed.nThis usually means the libguestfs
> appliance failed to start or crashed.nDo:n export
> LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1nand run the command
> again. For further information, read:n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfsnYou
> can also run \\\'libguestfs-test-tool\\\' and post the
> *complete* outputninto a bug report or message to the
> libguestfs mailing list.", "timestamp":
> "2020-10-11T22:05:08.397538670-04:00", "type": "error" }n\\\'
> err=b"virt-sparsify: error: libguestfs error: guestfs_launch
> failed.nThis usually means the libguestfs appliance failed to
> start or crashed.nDo:n export LIBGUESTFS_DEBUG=1
> LIBGUESTFS_TRACE=1nand run the command again. For further
> information, read:n
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfsnYou
> can also run \\\'libguestfs-test-tool\\\' and post the *complete*
> outputninto a bug report or message to the libguestfs mailing
> list.nnIf reporting bugs, run virt-sparsify with debugging
> enabled and include the ncomplete output:nn virt-sparsify
> -v -x [...]n"\',)'}}} (__init__:356)
>
> /var/log/messages has:
>
> Oct 11 22:04:51 lago-basic-suite-master-host-0 kvm[80601]: 1 guest now active
> Oct 11 22:05:06 lago-basic-suite-master-host-0 journal[80557]: Domain
> id=1 name='guestfs-hl0ntvn92rtkk2u0'
> uuid=05ea5a53-562f-49f8-a8ca-76b45c5325b4 is tainted: custom-argv
> Oct 11 22:05:06
