First off, I think the whole LLDB team can agree on the simple principle that tests that sometimes fail is not something we’re shooting for, so we’re on the same page with the general sentiment.
But, rather than talking in sweeping generalizations, let’s take this exact failure as an example. It was a socket attempting to connect. It uses general socket timeouts. If it’s using a default, it will typically be something like 30 seconds. Per my previous comment, this is not something we see fail with any kind of frequency. We just had a failure on it now. Is the solution to say we want 35 seconds? 1 minute? 2 minutes? We’re using end-to-end testing here, and therefore we are subject to how a real system behaves. If we crank that up, there could be a time where we overload the server further, and the socket facility just fails. What do you do then, claim the test as unworthy? The challenge we run into on LLDB is that, while we do have a growing number of unit tests that are strict input/output verification, far too much of a functioning debugger’s operations are not covered adequately by that. So we have a large body of end to end tests. Sometimes those fail under heavy load. Sometimes those issues are ours - we have hidden races that are only exposed when the machine is grinding to a halt. Sometimes it is that the OS fails to handle requests for resources that the debugger needs. So it isn’t as simple as saying “file a bug to make the LLDB test suite more reliable.” (I wish it were!) On the positive side, we have more resources going into improving our quality in the upcoming months, and we now have a dedicated quality engineer working with us. I expect this will help things as we move forward. And we’re looking to sink some of the system testing that currently exists end-to-end in LLDB to a more input-output style testing that we can do without needing the whole of LLDB (e.g. DWARF verification, DWARF type round-tripping through the compiler, etc. - Greg and Sean can speak more to what we’re doing there). The net effect is that we should be hitting less of this noise. -Todd On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev ( swift-lldb-...@swift.org) wrote: The amount of flakiness in the LLDB tests is disturbing... is there a high-level bug tracking improving this? It seems like it might be worth running a stress test of the tests on a loaded machine to try and shake out such problems. - Daniel On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev < swift-lldb-...@swift.org> wrote: On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com> wrote: On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev < swift-lldb-...@swift.org> wrote: It looks like this may be the packaging build: oss-swift-package-linux-ubuntu-15_10 It was this build: https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 It has since gone blue. I don’t think I recall ever seeing that test fail before in an intermittent way, but that might be what happened here. Given the failure mode, it does sound likely: File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr = self._sock.accept() timeout: timed out If we see that again, we may be able to tweak the socket timeout to allow for heavily loaded test CI. -Todd On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev < swift-lldb-...@swift.org> wrote: Where is this failure being reported? I’m not seeing anything on swift-3.0-branch or master on ci.swift.org. Kate Stone k8st...@apple.com Xcode Low Level Tools On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev < swift-lldb-...@swift.org> wrote: + swift-lldb-dev group. On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com> wrote: Failing in lldb: Command invoked: /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py --executable /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb --rerun-all-issues -C /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p TestStubReverseConnect.py /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=14:int Configuration: arch=x86_64 compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang ---------------------------------------------------------------------- Collected 2 tests ====================================================================== ERROR: test_reverse_connect_works_llgs (TestStubReverseConnect.TestStubReverseConnect) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", line 121, in wrapper func(*args, **kwargs) File "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", line 121, in wrapper func(*args, **kwargs) File "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", line 89, in test_reverse_connect_works_llgs self.reverse_connect_works() File "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", line 67, in reverse_connect_works (stub_socket, address) = self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr = self._sock.accept() timeout: timed out Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 ---------------------------------------------------------------------- Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 errors, 1 skipped, 0 expected failures, 0 unexpected successes) Session logs for test failures/errors/unexpected successes can be found in directory '2016-08-18-15_15_07' [TestStubReverseConnect.py FAILED] Command invoked: /usr/bin/python /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py --executable /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb --rerun-all-issues -C /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p TestStubReverseConnect.py /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=14:int - Daniel On Aug 18, 2016, at 9:18 PM, no-re...@swift.org wrote: [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] Build URL: https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ Project: oss-swift-package-linux-ubuntu-15_10 Date of build: Thu, 18 Aug 2016 20:26:52 -0700 Build duration: 51 min Identified problems: - Regression test failed: This build failed because a regression test in the test suite FAILed. Below is a list of all errors: - Indication 1 <https://ci.swift.org//job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> Changes - Commit *2ebab526bfd7a5ce6ef460bebee0399fe669612d* by *xi_ge:* [FixCode] Apply coercion fixits on return statement and initialization - *edit*: test/FixCode/fixits-apply-objc.swift - *edit*: test/FixCode/fixits-apply-objc.swift.result - *edit*: lib/Sema/CSDiag.cpp - *edit*: test/FixCode/fixits-apply.swift.result - - Commit *6447a2d16edcfbceab940d58a321b3ef70525f1a* by *daniel_dunbar:* [Basic] Update Thread shim to match changes on Linux side. - *edit*: Sources/Basic/Thread.swift _______________________________________________ swift-lldb-dev mailing list swift-lldb-...@swift.org https://lists.swift.org/mailman/listinfo/swift-lldb-dev _______________________________________________ swift-lldb-dev mailing list swift-lldb-...@swift.org https://lists.swift.org/mailman/listinfo/swift-lldb-dev _______________________________________________ swift-lldb-dev mailing list swift-lldb-...@swift.org https://lists.swift.org/mailman/listinfo/swift-lldb-dev Thanks, *- Enrico* 📩 egranata@ <about:blank> <about:blank>.com <about:blank> ☎️ 27683 _______________________________________________ swift-lldb-dev mailing list swift-lldb-...@swift.org https://lists.swift.org/mailman/listinfo/swift-lldb-dev _______________________________________________ swift-lldb-dev mailing list swift-lldb-...@swift.org https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
