Re: biosbits test failing on origin/master
On Wed, Nov 16, 2022 at 08:34:00AM +0530, Ani Sinha wrote: > On Wed, Nov 16, 2022 at 12:18 AM John Snow wrote: > > > > On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha wrote: > > > > > > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > > > > > > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > > > > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow > > > > > > > wrote: > > > > > > > > > > > > > > > > Hiya, on today's origin/master > > > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that > > > > > > > > "make > > > > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > > > > development machine: > > > > > > > > > > > > > > > > (001/193) > > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > > FAIL: True is not false : The VM seems to have failed to > > > > > > > > shutdown in > > > > > > > > time (83.65 s) > > > > > > > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > > > > > > > In my test environment it does pass. > > > > > > > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > > > > Fetching asset from > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > > > > JOB LOG: > > > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > > > > (1/1) > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > PASS (57.57 s) > > > > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | > > > > > > > INTERRUPT 0 > > > > > > > | CANCEL 0 > > > > > > > JOB TIME : 63.82 s > > > > > > > > > > > > > > However, I have seen that on certain slower test machines or when > > > > > > > run > > > > > > > within a virtual machine, the test can take longer to complete > > > > > > > and 60 > > > > > > > secs may not always be enough. In those cases raising the maximum > > > > > > > completion time to 90 secs helps. Perhaps you can try this and > > > > > > > let me > > > > > > > know if it helps: > > > > > > > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > > > > have an invocation to share that exists outside of the avocado > > > > > > machinery > > > > > > > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > > > > that was used to run the test. You also need to comment out the line > > > > > > shutil.rmtree(self._workDir) > > > > > in tearDown() so that the iso is not cleaned up. > > > > > > > > Maybe I will send out this patch once we have figured out what is > > > > going on with your environment: > > > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > > > > > > > I ran the test on another box sitting in my office running centos7.9 > > > > and it passed as well. > > > > > > For the records, > > > I ran make check-avocado on my Ubuntu 222.04 laptop: > > > https://pastebin.com/0ZKEEQds > > > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4 > > > > As a question: Is it necessary to implement your own timeout here? > > What's wrong with relying on Avocado's timeout? > > When I wrote the test I was not aware of the avocado timeout. Hence > implemented mine. However, I kind of think that timing out from the > test itself rather than from the framework provides an opportunity to > provide more meaningful information to the user when the timeout > happens. Of Course we can improve upon the current "FAIL: True is not > false" assertion message. > > > My hunch is that you'll get greater flexibility by leaning into the > > tool suite's configuration instead of hardcoding your own... > > > > For what it's worth, I am now trying to run this test manually by doing: > > > > > time ./qemu-system-x86_64 -cdrom > > > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto > > > > This is not a quick test: > > > > > > Executed in 86.50 secsfish external > >usr time 86.57 secs0.00 micros 86.57 secs > >sys time0.30 secs 903.00 micros0.29 secs > > > > > > This isn't the *most* cutting edge machine, but it's a Intel(R) > > Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage. > > Is your machine really passing this test in under 60 seconds flat? > > Yes it is both on my 3 year old lenovo laptop with 16 Gib memory: Note gitlab.com public CI shared runners are *very* small VM instances. IIUC, currently we can expect the runner to have as little as 1 vCPU, and 3.75 GB of RAM. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org
Re: biosbits test failing on origin/master
On Wed, Nov 16, 2022 at 3:22 AM John Snow wrote: > > On Tue, Nov 15, 2022 at 1:47 PM John Snow wrote: > > > > On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha wrote: > > > > > > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > > > > > > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > > > > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow > > > > > > > wrote: > > > > > > > > > > > > > > > > Hiya, on today's origin/master > > > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that > > > > > > > > "make > > > > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > > > > development machine: > > > > > > > > > > > > > > > > (001/193) > > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > > FAIL: True is not false : The VM seems to have failed to > > > > > > > > shutdown in > > > > > > > > time (83.65 s) > > > > > > > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > > > > > > > In my test environment it does pass. > > > > > > > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > > > > Fetching asset from > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > > > > JOB LOG: > > > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > > > > (1/1) > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > PASS (57.57 s) > > > > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | > > > > > > > INTERRUPT 0 > > > > > > > | CANCEL 0 > > > > > > > JOB TIME : 63.82 s > > > > > > > > > > > > > > However, I have seen that on certain slower test machines or when > > > > > > > run > > > > > > > within a virtual machine, the test can take longer to complete > > > > > > > and 60 > > > > > > > secs may not always be enough. In those cases raising the maximum > > > > > > > completion time to 90 secs helps. Perhaps you can try this and > > > > > > > let me > > > > > > > know if it helps: > > > > > > > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > > > > have an invocation to share that exists outside of the avocado > > > > > > machinery > > > > > > > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > > > > that was used to run the test. You also need to comment out the line > > > > > > shutil.rmtree(self._workDir) > > > > > in tearDown() so that the iso is not cleaned up. > > > > > > > > Maybe I will send out this patch once we have figured out what is > > > > going on with your environment: > > > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > > > > > > > I ran the test on another box sitting in my office running centos7.9 > > > > and it passed as well. > > > > > > For the records, > > > I ran make check-avocado on my Ubuntu 222.04 laptop: > > > https://pastebin.com/0ZKEEQds > > > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4 > > > > As a question: Is it necessary to implement your own timeout here? > > What's wrong with relying on Avocado's timeout? > > My hunch is that you'll get greater flexibility by leaning into the > > tool suite's configuration instead of hardcoding your own... > > > > For what it's worth, I am now trying to run this test manually by doing: > > > > > time ./qemu-system-x86_64 -cdrom > > > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto > > > > This is not a quick test: > > > > > > Executed in 86.50 secsfish external > >usr time 86.57 secs0.00 micros 86.57 secs > >sys time0.30 secs 903.00 micros0.29 secs > > > > > > This isn't the *most* cutting edge machine, but it's a Intel(R) > > Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage. > > Is your machine really passing this test in under 60 seconds flat? > > > > FWIW, I sent a patch to just disable the test-specific timeout. I > figure you had a good reason for putting it there, but I suppose > you'll tell me tomorrow what that reason was :) We need to wait until the VM terminates. The reason for providing the timeout was that (a) I figure it is a bad idea to wait indefinitely in case the VM gets wedged or stuck. (b) I was not aware of the framework level timeout mechanism. I am ok if we want to get rid of the internal timeout. However, we need to wait for the VM to terminate. > > I couldn't get the test to finish in under 60 seconds on this machine, > but I did manage to get it to finish in under 60 on my ... 16 core AMD > 5950X. Just needed some very heavy firepower, I guess! > > --js >
Re: biosbits test failing on origin/master
On Wed, Nov 16, 2022 at 12:18 AM John Snow wrote: > > On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha wrote: > > > > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > > > > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > > > > > > > > > Hiya, on today's origin/master > > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > > > development machine: > > > > > > > > > > > > > > (001/193) > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > FAIL: True is not false : The VM seems to have failed to shutdown > > > > > > > in > > > > > > > time (83.65 s) > > > > > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > > > > > In my test environment it does pass. > > > > > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > > > Fetching asset from > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > > > JOB LOG: > > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > > > (1/1) > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > PASS (57.57 s) > > > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | > > > > > > INTERRUPT 0 > > > > > > | CANCEL 0 > > > > > > JOB TIME : 63.82 s > > > > > > > > > > > > However, I have seen that on certain slower test machines or when > > > > > > run > > > > > > within a virtual machine, the test can take longer to complete and > > > > > > 60 > > > > > > secs may not always be enough. In those cases raising the maximum > > > > > > completion time to 90 secs helps. Perhaps you can try this and let > > > > > > me > > > > > > know if it helps: > > > > > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > > > have an invocation to share that exists outside of the avocado > > > > > machinery > > > > > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > > > that was used to run the test. You also need to comment out the line > > > > > shutil.rmtree(self._workDir) > > > > in tearDown() so that the iso is not cleaned up. > > > > > > Maybe I will send out this patch once we have figured out what is > > > going on with your environment: > > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > > > > > I ran the test on another box sitting in my office running centos7.9 > > > and it passed as well. > > > > For the records, > > I ran make check-avocado on my Ubuntu 222.04 laptop: > > https://pastebin.com/0ZKEEQds > > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4 > > As a question: Is it necessary to implement your own timeout here? > What's wrong with relying on Avocado's timeout? When I wrote the test I was not aware of the avocado timeout. Hence implemented mine. However, I kind of think that timing out from the test itself rather than from the framework provides an opportunity to provide more meaningful information to the user when the timeout happens. Of Course we can improve upon the current "FAIL: True is not false" assertion message. > My hunch is that you'll get greater flexibility by leaning into the > tool suite's configuration instead of hardcoding your own... > > For what it's worth, I am now trying to run this test manually by doing: > > > time ./qemu-system-x86_64 -cdrom > > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto > > This is not a quick test: > > > Executed in 86.50 secsfish external >usr time 86.57 secs0.00 micros 86.57 secs >sys time0.30 secs 903.00 micros0.29 secs > > > This isn't the *most* cutting edge machine, but it's a Intel(R) > Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage. > Is your machine really passing this test in under 60 seconds flat? Yes it is both on my 3 year old lenovo laptop with 16 Gib memory: Vendor ID: GenuineIntel Model name:Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz CPU family: 6 Model: 142 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Stepping:12 CPU max MHz: 4100. CPU min MHz: 400. as well as on a desktop machine with 32 Gib of memory: Vendor ID: GenuineIntel CPU family:6 Model: 158 Model name:Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz Stepping: 13 CPU MHz:
Re: biosbits test failing on origin/master
On Tue, Nov 15, 2022 at 1:47 PM John Snow wrote: > > On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha wrote: > > > > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > > > > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > > > > > > > > > Hiya, on today's origin/master > > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > > > development machine: > > > > > > > > > > > > > > (001/193) > > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > > FAIL: True is not false : The VM seems to have failed to shutdown > > > > > > > in > > > > > > > time (83.65 s) > > > > > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > > > > > In my test environment it does pass. > > > > > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > > > Fetching asset from > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > > > JOB LOG: > > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > > > (1/1) > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > PASS (57.57 s) > > > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | > > > > > > INTERRUPT 0 > > > > > > | CANCEL 0 > > > > > > JOB TIME : 63.82 s > > > > > > > > > > > > However, I have seen that on certain slower test machines or when > > > > > > run > > > > > > within a virtual machine, the test can take longer to complete and > > > > > > 60 > > > > > > secs may not always be enough. In those cases raising the maximum > > > > > > completion time to 90 secs helps. Perhaps you can try this and let > > > > > > me > > > > > > know if it helps: > > > > > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > > > have an invocation to share that exists outside of the avocado > > > > > machinery > > > > > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > > > that was used to run the test. You also need to comment out the line > > > > > shutil.rmtree(self._workDir) > > > > in tearDown() so that the iso is not cleaned up. > > > > > > Maybe I will send out this patch once we have figured out what is > > > going on with your environment: > > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > > > > > I ran the test on another box sitting in my office running centos7.9 > > > and it passed as well. > > > > For the records, > > I ran make check-avocado on my Ubuntu 222.04 laptop: > > https://pastebin.com/0ZKEEQds > > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4 > > As a question: Is it necessary to implement your own timeout here? > What's wrong with relying on Avocado's timeout? > My hunch is that you'll get greater flexibility by leaning into the > tool suite's configuration instead of hardcoding your own... > > For what it's worth, I am now trying to run this test manually by doing: > > > time ./qemu-system-x86_64 -cdrom > > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto > > This is not a quick test: > > > Executed in 86.50 secsfish external >usr time 86.57 secs0.00 micros 86.57 secs >sys time0.30 secs 903.00 micros0.29 secs > > > This isn't the *most* cutting edge machine, but it's a Intel(R) > Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage. > Is your machine really passing this test in under 60 seconds flat? > FWIW, I sent a patch to just disable the test-specific timeout. I figure you had a good reason for putting it there, but I suppose you'll tell me tomorrow what that reason was :) I couldn't get the test to finish in under 60 seconds on this machine, but I did manage to get it to finish in under 60 on my ... 16 core AMD 5950X. Just needed some very heavy firepower, I guess! --js
Re: biosbits test failing on origin/master
On Tue, Nov 15, 2022 at 9:31 AM Ani Sinha wrote: > > On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > > > > > > > Hiya, on today's origin/master > > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > > development machine: > > > > > > > > > > > > (001/193) > > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > > FAIL: True is not false : The VM seems to have failed to shutdown in > > > > > > time (83.65 s) > > > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > > > In my test environment it does pass. > > > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > > Fetching asset from > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > > JOB LOG: > > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > PASS (57.57 s) > > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > > > > > | CANCEL 0 > > > > > JOB TIME : 63.82 s > > > > > > > > > > However, I have seen that on certain slower test machines or when run > > > > > within a virtual machine, the test can take longer to complete and 60 > > > > > secs may not always be enough. In those cases raising the maximum > > > > > completion time to 90 secs helps. Perhaps you can try this and let me > > > > > know if it helps: > > > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > > have an invocation to share that exists outside of the avocado > > > > machinery > > > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > > that was used to run the test. You also need to comment out the line > > > > shutil.rmtree(self._workDir) > > > in tearDown() so that the iso is not cleaned up. > > > > Maybe I will send out this patch once we have figured out what is > > going on with your environment: > > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > > > I ran the test on another box sitting in my office running centos7.9 > > and it passed as well. > > For the records, > I ran make check-avocado on my Ubuntu 222.04 laptop: > https://pastebin.com/0ZKEEQds > On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4 As a question: Is it necessary to implement your own timeout here? What's wrong with relying on Avocado's timeout? My hunch is that you'll get greater flexibility by leaning into the tool suite's configuration instead of hardcoding your own... For what it's worth, I am now trying to run this test manually by doing: > time ./qemu-system-x86_64 -cdrom > /var/tmp/acpi-bits-b_br0ch8.tmp/bits-2020.iso -icount auto This is not a quick test: Executed in 86.50 secsfish external usr time 86.57 secs0.00 micros 86.57 secs sys time0.30 secs 903.00 micros0.29 secs This isn't the *most* cutting edge machine, but it's a Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz with 32GB of memory and NVME storage. Is your machine really passing this test in under 60 seconds flat? --js
Re: biosbits test failing on origin/master
On Tue, Nov 15, 2022 at 3:36 PM Ani Sinha wrote: > > On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > > > > > Hiya, on today's origin/master > > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > > > check-avocado" is failing on the new biosbits test on my local > > > > > development machine: > > > > > > > > > > (001/193) > > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > > FAIL: True is not false : The VM seems to have failed to shutdown in > > > > > time (83.65 s) > > > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > > > In my test environment it does pass. > > > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > > Fetching asset from > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > > JOB LOG: > > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > PASS (57.57 s) > > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > > > > | CANCEL 0 > > > > JOB TIME : 63.82 s > > > > > > > > However, I have seen that on certain slower test machines or when run > > > > within a virtual machine, the test can take longer to complete and 60 > > > > secs may not always be enough. In those cases raising the maximum > > > > completion time to 90 secs helps. Perhaps you can try this and let me > > > > know if it helps: > > > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > > have an invocation to share that exists outside of the avocado > > > machinery > > > > If you pass V=1 in the environment then it dumps the QEMU command line > > that was used to run the test. You also need to comment out the line > > > shutil.rmtree(self._workDir) > > in tearDown() so that the iso is not cleaned up. > > Maybe I will send out this patch once we have figured out what is > going on with your environment: > https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e > > I ran the test on another box sitting in my office running centos7.9 > and it passed as well. For the records, I ran make check-avocado on my Ubuntu 222.04 laptop: https://pastebin.com/0ZKEEQds On a separate centos 7.9 box (fairly new) : https://pastebin.com/QWLGDbp4
Re: biosbits test failing on origin/master
On Tue, Nov 15, 2022 at 9:07 AM Ani Sinha wrote: > > On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote: > > > > On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > > > Hiya, on today's origin/master > > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > > check-avocado" is failing on the new biosbits test on my local > > > > development machine: > > > > > > > > (001/193) > > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > > FAIL: True is not false : The VM seems to have failed to shutdown in > > > > time (83.65 s) > > > > > > > > Is this a known issue, or should I begin to investigate it? > > > > > > In my test environment it does pass. > > > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > > Fetching asset from > > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > > JOB LOG: > > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > PASS (57.57 s) > > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > > > | CANCEL 0 > > > JOB TIME : 63.82 s > > > > > > However, I have seen that on certain slower test machines or when run > > > within a virtual machine, the test can take longer to complete and 60 > > > secs may not always be enough. In those cases raising the maximum > > > completion time to 90 secs helps. Perhaps you can try this and let me > > > know if it helps: > > > > Hmm - I'm running on a fairly modern machine and not in a VM. Do you > > have an invocation to share that exists outside of the avocado > > machinery > > If you pass V=1 in the environment then it dumps the QEMU command line > that was used to run the test. You also need to comment out the line > > shutil.rmtree(self._workDir) > in tearDown() so that the iso is not cleaned up. Maybe I will send out this patch once we have figured out what is going on with your environment: https://gitlab.com/anisinha/qemu/-/commit/5e8c629fdecc7cb650e4acaad8a8fcc2b248434e I ran the test on another box sitting in my office running centos7.9 and it passed as well.
Re: biosbits test failing on origin/master
On Tue, Nov 15, 2022 at 5:13 AM John Snow wrote:
>
> On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote:
> >
> > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote:
> > >
> > > Hiya, on today's origin/master
> > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make
> > > check-avocado" is failing on the new biosbits test on my local
> > > development machine:
> > >
> > > (001/193) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits:
> > > FAIL: True is not false : The VM seems to have failed to shutdown in
> > > time (83.65 s)
> > >
> > > Is this a known issue, or should I begin to investigate it?
> >
> > In my test environment it does pass.
> >
> > $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> > Fetching asset from
> > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9
> > JOB LOG:
> > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log
> > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits:
> > PASS (57.57 s)
> > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
> > | CANCEL 0
> > JOB TIME : 63.82 s
> >
> > However, I have seen that on certain slower test machines or when run
> > within a virtual machine, the test can take longer to complete and 60
> > secs may not always be enough. In those cases raising the maximum
> > completion time to 90 secs helps. Perhaps you can try this and let me
> > know if it helps:
>
> Hmm - I'm running on a fairly modern machine and not in a VM. Do you
> have an invocation to share that exists outside of the avocado
> machinery
If you pass V=1 in the environment then it dumps the QEMU command line
that was used to run the test. You also need to comment out the line
> shutil.rmtree(self._workDir)
in tearDown() so that the iso is not cleaned up.
diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py
index a67d30d583..2060e3b84f 100644
--- a/tests/avocado/acpi-bits.py
+++ b/tests/avocado/acpi-bits.py
@@ -357,7 +357,7 @@ def tearDown(self):
if self._vm:
self.assertFalse(not self._vm.is_running)
self.logger.info('removing the work directory %s', self._workDir)
-shutil.rmtree(self._workDir)
+# shutil.rmtree(self._workDir)
super().tearDown()
def test_acpi_smbios_bits(self):
while you are at it, it might makes sense to check the vnc for the VM
to see what it is doing.
where I could test this individually and see how long it
> might take to complete if I just let it run? I am worried that it's
> getting wedged instead of just taking a long time, but it's hard to
> tell.
>
> --js
>
> >
> > diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py
> > index 8745a58a76..b11fe39350 100644
> > --- a/tests/avocado/acpi-bits.py
> > +++ b/tests/avocado/acpi-bits.py
> > @@ -385,8 +385,9 @@ def test_acpi_smbios_bits(self):
> > self._vm.launch()
> > # biosbits has been configured to run all the specified test suites
> > # in batch mode and then automatically initiate a vm shutdown.
> > -# sleep for maximum of one minute
> > -max_sleep_time = time.monotonic() + 60
> > +# sleep for a maximum of one and half minutes to accommodate
> > running this
> > +# even on slower machines.
> > +max_sleep_time = time.monotonic() + 90
> > while self._vm.is_running() and time.monotonic() < max_sleep_time:
> > time.sleep(1)
> >
>
Re: biosbits test failing on origin/master
On Thu, Nov 10, 2022 at 11:22 PM Ani Sinha wrote: > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > Hiya, on today's origin/master > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > check-avocado" is failing on the new biosbits test on my local > > development machine: > > > > (001/193) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > FAIL: True is not false : The VM seems to have failed to shutdown in > > time (83.65 s) > > > > Is this a known issue, or should I begin to investigate it? > > In my test environment it does pass. > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > JOB LOG: > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > PASS (57.57 s) > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > | CANCEL 0 > JOB TIME : 63.82 s > > However, I have seen that on certain slower test machines or when run > within a virtual machine, the test can take longer to complete and 60 > secs may not always be enough. In those cases raising the maximum > completion time to 90 secs helps. Perhaps you can try this and let me > know if it helps: Hmm - I'm running on a fairly modern machine and not in a VM. Do you have an invocation to share that exists outside of the avocado machinery where I could test this individually and see how long it might take to complete if I just let it run? I am worried that it's getting wedged instead of just taking a long time, but it's hard to tell. --js > > diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py > index 8745a58a76..b11fe39350 100644 > --- a/tests/avocado/acpi-bits.py > +++ b/tests/avocado/acpi-bits.py > @@ -385,8 +385,9 @@ def test_acpi_smbios_bits(self): > self._vm.launch() > # biosbits has been configured to run all the specified test suites > # in batch mode and then automatically initiate a vm shutdown. > -# sleep for maximum of one minute > -max_sleep_time = time.monotonic() + 60 > +# sleep for a maximum of one and half minutes to accommodate > running this > +# even on slower machines. > +max_sleep_time = time.monotonic() + 90 > while self._vm.is_running() and time.monotonic() < max_sleep_time: > time.sleep(1) >
Re: biosbits test failing on origin/master
On Fri, Nov 11, 2022 at 4:39 PM Ani Sinha wrote: > > On Fri, Nov 11, 2022 at 9:52 AM Ani Sinha wrote: > > > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > > > Hiya, on today's origin/master > > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > > check-avocado" is failing on the new biosbits test on my local > > > development machine: > > > > > > (001/193) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > > FAIL: True is not false : The VM seems to have failed to shutdown in > > > time (83.65 s) > > > > > > Is this a known issue, or should I begin to investigate it? > > > > In my test environment it does pass. > > > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > Fetching asset from > > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > > JOB LOG: > > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > PASS (57.57 s) > > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > > | CANCEL 0 > > JOB TIME : 63.82 s > > > > However, I have seen that on certain slower test machines or when run > > within a virtual machine, the test can take longer to complete and 60 > > secs may not always be enough. > > Here is an interesting data point. I re-ran on a Centos 8 VM running > on Ubuntu bare metal. Just like the Ubuntu bare metal, it passed fine. > Next, I tried to run it on a Centos 7.9.2009 VM running on the same > Ubuntu bare metal. I was able to reproduce this consistently. So I did > some digging, gdb into qemu and it looked like the VM was just slow > but not stuck. So I timed the QEMU command line that was being used in > the test using the same iso that was being generated by the test. Here > it is: > > # time ./qemu-system-x86_64 -display none -vga none -chardev > file,path=/var/tmp/debugcon-log.txt,id=debugcon -device > isa-debugcon,iobase=0x403,chardev=debugcon -cdrom > /var/tmp/acpi-bits-oogd8wp9.tmp/bits-2020.iso > > real 2m34.052s > user 2m33.858s > sys 0m0.467s > > On bare metal Ubuntu, I see this: > > $ time ./qemu-system-x86_64 -display none -vga none -chardev > file,path=/var/tmp/debugcon-log.txt,id=debugcon -device > isa-debugcon,iobase=0x403,chardev=debugcon -cdrom ~/temp/bits-2020.iso > > real 1m15.318s > user 1m15.136s > sys 0m0.345s > > With "-icount auto" added in the command line, both bare metal and VM > environments were slower. However, we need this command line for some > latency tests. > > avocado framework has a 2 min timeout for any test. Just realized that this value is coming from https://gitlab.com/anisinha/qemu/-/blob/master/tests/avocado/avocado_qemu/__init__.py#L232 and can be overiden. So I will override this to 200 seconds. If we still see timeouts, we can raise this further by a bit more. Thanks for bringing this to my attention. This would > definitely time out even without my internal timeout check. I verified > that the test indeed passed by looking into the debug con logs pushed > out by the test framework. If the system is just slow/overloaded I am > not sure what we can do. > > In those cases raising the maximum > > completion time to 90 secs helps. Perhaps you can try this and let me > > know if it helps: > > Maybe I will make the timeout 90 secs and hope that this will be enough. > > > > > diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py > > index 8745a58a76..b11fe39350 100644 > > --- a/tests/avocado/acpi-bits.py > > +++ b/tests/avocado/acpi-bits.py > > @@ -385,8 +385,9 @@ def test_acpi_smbios_bits(self): > > self._vm.launch() > > # biosbits has been configured to run all the specified test suites > > # in batch mode and then automatically initiate a vm shutdown. > > -# sleep for maximum of one minute > > -max_sleep_time = time.monotonic() + 60 > > +# sleep for a maximum of one and half minutes to accommodate > > running this > > +# even on slower machines. > > +max_sleep_time = time.monotonic() + 90 > > while self._vm.is_running() and time.monotonic() < max_sleep_time: > > time.sleep(1)
Re: biosbits test failing on origin/master
On Fri, Nov 11, 2022 at 9:52 AM Ani Sinha wrote: > > On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > > > Hiya, on today's origin/master > > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > > check-avocado" is failing on the new biosbits test on my local > > development machine: > > > > (001/193) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > > FAIL: True is not false : The VM seems to have failed to shutdown in > > time (83.65 s) > > > > Is this a known issue, or should I begin to investigate it? > > In my test environment it does pass. > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 > JOB LOG: > /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > PASS (57.57 s) > RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 > | CANCEL 0 > JOB TIME : 63.82 s > > However, I have seen that on certain slower test machines or when run > within a virtual machine, the test can take longer to complete and 60 > secs may not always be enough. Here is an interesting data point. I re-ran on a Centos 8 VM running on Ubuntu bare metal. Just like the Ubuntu bare metal, it passed fine. Next, I tried to run it on a Centos 7.9.2009 VM running on the same Ubuntu bare metal. I was able to reproduce this consistently. So I did some digging, gdb into qemu and it looked like the VM was just slow but not stuck. So I timed the QEMU command line that was being used in the test using the same iso that was being generated by the test. Here it is: # time ./qemu-system-x86_64 -display none -vga none -chardev file,path=/var/tmp/debugcon-log.txt,id=debugcon -device isa-debugcon,iobase=0x403,chardev=debugcon -cdrom /var/tmp/acpi-bits-oogd8wp9.tmp/bits-2020.iso real 2m34.052s user 2m33.858s sys 0m0.467s On bare metal Ubuntu, I see this: $ time ./qemu-system-x86_64 -display none -vga none -chardev file,path=/var/tmp/debugcon-log.txt,id=debugcon -device isa-debugcon,iobase=0x403,chardev=debugcon -cdrom ~/temp/bits-2020.iso real 1m15.318s user 1m15.136s sys 0m0.345s With "-icount auto" added in the command line, both bare metal and VM environments were slower. However, we need this command line for some latency tests. avocado framework has a 2 min timeout for any test. This would definitely time out even without my internal timeout check. I verified that the test indeed passed by looking into the debug con logs pushed out by the test framework. If the system is just slow/overloaded I am not sure what we can do. In those cases raising the maximum > completion time to 90 secs helps. Perhaps you can try this and let me > know if it helps: Maybe I will make the timeout 90 secs and hope that this will be enough. > > diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py > index 8745a58a76..b11fe39350 100644 > --- a/tests/avocado/acpi-bits.py > +++ b/tests/avocado/acpi-bits.py > @@ -385,8 +385,9 @@ def test_acpi_smbios_bits(self): > self._vm.launch() > # biosbits has been configured to run all the specified test suites > # in batch mode and then automatically initiate a vm shutdown. > -# sleep for maximum of one minute > -max_sleep_time = time.monotonic() + 60 > +# sleep for a maximum of one and half minutes to accommodate > running this > +# even on slower machines. > +max_sleep_time = time.monotonic() + 90 > while self._vm.is_running() and time.monotonic() < max_sleep_time: > time.sleep(1)
Re: biosbits test failing on origin/master
On Thu, Nov 10, 2022 at 11:37 PM John Snow wrote: > > Hiya, on today's origin/master > (2ccad61746ca7de5dd3e25146062264387e43bd4) I'm finding that "make > check-avocado" is failing on the new biosbits test on my local > development machine: > > (001/193) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: > FAIL: True is not false : The VM seems to have failed to shutdown in > time (83.65 s) > > Is this a known issue, or should I begin to investigate it? In my test environment it does pass. $ ./tests/venv/bin/avocado run -t acpi tests/avocado Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : 35726df7d3c2e0f41847822620c78195ba45b9b9 JOB LOG: /home/anisinha/avocado/job-results/job-2022-11-11T09.42-35726df/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (57.57 s) RESULTS: PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 63.82 s However, I have seen that on certain slower test machines or when run within a virtual machine, the test can take longer to complete and 60 secs may not always be enough. In those cases raising the maximum completion time to 90 secs helps. Perhaps you can try this and let me know if it helps: diff --git a/tests/avocado/acpi-bits.py b/tests/avocado/acpi-bits.py index 8745a58a76..b11fe39350 100644 --- a/tests/avocado/acpi-bits.py +++ b/tests/avocado/acpi-bits.py @@ -385,8 +385,9 @@ def test_acpi_smbios_bits(self): self._vm.launch() # biosbits has been configured to run all the specified test suites # in batch mode and then automatically initiate a vm shutdown. -# sleep for maximum of one minute -max_sleep_time = time.monotonic() + 60 +# sleep for a maximum of one and half minutes to accommodate running this +# even on slower machines. +max_sleep_time = time.monotonic() + 90 while self._vm.is_running() and time.monotonic() < max_sleep_time: time.sleep(1)
