On Mon, Nov 9, 2020 at 11:03 AM Richard W.M. Jones <rjo...@redhat.com> wrote: > > On Mon, Nov 09, 2020 at 09:16:33AM +0200, Yedidyah Bar David wrote: > > On Wed, Oct 14, 2020 at 8:54 AM Yedidyah Bar David <d...@redhat.com> wrote: > > > > > > On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjo...@redhat.com> > > > wrote: > > > > > > > > On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote: > > > > > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones > > > > > <rjo...@redhat.com> wrote: > > > > > > > > > > > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote: > > > > > > > I think this is the right solution - when virt-something tool > > > > > > > fails, > > > > > > > it should log the reason for the failure - the error that caused > > > > > > > the > > > > > > > tool to fail. I'm not sure this is easy to do as the failing code > > > > > > > run inside a special VM. Maybe the code running in the VM should > > > > > > > log > > > > > > > the output in a machine readable way, so once an error is detected > > > > > > > virt-something can report the error as the reason, without running > > > > > > > in debug mode. > > > > > > > > > > > > All the virt-* tools that I've written have a non-zero exit code and > > > > > > print an error message on stderr when they fail. Errors from inside > > > > > > the appliance are propagated to the library and thence to the tool > > > > > > correctly. > > > > > > > > > > > > I think the best thing to do is: > > > > > > > > > > > > - spool up stdout + stderr from the tool > > > > > > > > > > > > - if the exit code != 0, save the spooled output for analysis > > > > > > > > > > > > - if the exit code == 0, discard it (or keep it if you like) > > > > > > > > > > This is what we already do, and the result is not helpful. If you look > > > > > at the log message in the previous message, basically the only > > > > > info about the error is: > > > > > > > > > > libguestfs error: guestfs_launch failed > > > > > > > > > > I don't see what we can do with this error message. > > > > > > > > Right, so in this particular instance the error message would tell us > > > > that you should run libguestfs-test-tool because your qemu/kernel/etc > > > > is broken in some way :-/ > > > > > > > > There's not a particularly good answer here if you don't want to ever > > > > use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run > > > > libguestfs-test-tool if you see any error which matches the substring > > > > /guestfs_launch/ ? > > > > > > Another (orthogonal?) option: > > > > > > Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err > > > (e.g. some other file descriptor, or to a file passed via env or > > > whatever). > > > This way, it might make sense for vdsm to always pass these vars, continue > > > logging all stdout/err, and log/keep debug/trace logs only on errors. > > > > This now happened again: > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/ > > > > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/565/artifact/exported-artifacts/test_logs/basic-suite-master/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log > > > > 2020-11-09 01:05:42,031-0500 INFO (jsonrpc/4) [api.host] FINISH > > getAllVmIoTunePolicies return={'status': {'code': 0, 'message': > > 'Done'}, 'io_tune_policies_dict': > > {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [], > > 'current_values': [{'name': 'vda', 'path': > > '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80', > > 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, > > 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, > > 'read_iops_sec': 0}}]}}} from=::1,34002 (api:54) > > 2020-11-09 01:05:42,038-0500 DEBUG (jsonrpc/4) [jsonrpc.JsonRpcServer] > > Return 'Host.getAllVmIoTunePolicies' in bridge with > > {'c189ecb3-8f2e-4726-8766-7d2d9b514687': {'policy': [], > > 'current_values': [{'name': 'vda', 'path': > > '/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/1d093232-d41e-483f-a915-62f8db3c972f/images/e7ee6417-b319-4d84-81a5-5d77cbce2385/710d2c10-e6b7-4d16-bd37-50a9d4e14a80', > > 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, > > 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, > > 'read_iops_sec': 0}}]}} (__init__:360) > > 2020-11-09 01:05:42,435-0500 DEBUG (tasks/3) [common.commands] FAILED: > > <err> = b"virt-sparsify: error: libguestfs error: guestfs_launch > > failed.\nThis usually means the libguestfs appliance failed to start > > or crashed.\nDo:\n export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\nand > > run the command again. For further information, read:\n > > http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\nYou can > > also run 'libguestfs-test-tool' and post the *complete* output\ninto a > > bug report or message to the libguestfs mailing list.\n\nIf reporting > > bugs, run virt-sparsify with debugging enabled and include the > > \ncomplete output:\n\n virt-sparsify -v -x [...]\n"; <rc> = 1 > > (commands:98) > > > > I suggest that if we have come to a dead-end and no-one has any clue, then > > we either patch something (vdsm?) to allow getting more information if this > > happens again, or open a bug for further discussion/prioritization. > > You definitely need to run libguestfs-test-tool from vdsm if we're > going to get any further with this.
Now filed [1] on vdsm. Feel free to comment there about what else we should do to get more information. Thanks! [1] https://bugzilla.redhat.com/show_bug.cgi?id=1895843 Best regards, > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-p2v converts physical machines to virtual machines. Boot with a > live CD or over the network (PXE) and turn machines into KVM guests. > http://libguestfs.org/virt-v2v > -- Didi _______________________________________________ Libguestfs mailing list Libguestfs@redhat.com https://www.redhat.com/mailman/listinfo/libguestfs
