> On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fi...@gmail.com> wrote: > > First off, I think the whole LLDB team can agree on the simple principle that > tests that sometimes fail is not something we’re shooting for, so we’re on > the same page with the general sentiment. > > But, rather than talking in sweeping generalizations, let’s take this exact > failure as an example. It was a socket attempting to connect. It uses > general socket timeouts. If it’s using a default, it will typically be > something like 30 seconds. Per my previous comment, this is not something we > see fail with any kind of frequency. We just had a failure on it now. Is > the solution to say we want 35 seconds? 1 minute? 2 minutes? We’re using > end-to-end testing here, and therefore we are subject to how a real system > behaves. If we crank that up, there could be a time where we overload the > server further, and the socket facility just fails. What do you do then, > claim the test as unworthy? > > The challenge we run into on LLDB is that, while we do have a growing number > of unit tests that are strict input/output verification, far too much of a > functioning debugger’s operations are not covered adequately by that. So we > have a large body of end to end tests. Sometimes those fail under heavy > load. Sometimes those issues are ours - we have hidden races that are only > exposed when the machine is grinding to a halt. Sometimes it is that the OS > fails to handle requests for resources that the debugger needs. > > So it isn’t as simple as saying “file a bug to make the LLDB test suite more > reliable.” (I wish it were!)
This is specifically why I proposed a concrete strategy like running a stress test of the test suite. That is actionable, and should shake some bugs I would hope? Another actionable strategies (although one I'm not a big fan of) is enhance the test suite to automatically re-attempt flaky tests. > > On the positive side, we have more resources going into improving our quality > in the upcoming months, and we now have a dedicated quality engineer working > with us. I expect this will help things as we move forward. And we’re > looking to sink some of the system testing that currently exists end-to-end > in LLDB to a more input-output style testing that we can do without needing > the whole of LLDB (e.g. DWARF verification, DWARF type round-tripping through > the compiler, etc. - Greg and Sean can speak more to what we’re doing there). Great! - Daniel > > The net effect is that we should be hitting less of this noise. > > -Todd > On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev > (swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>) wrote: > >> The amount of flakiness in the LLDB tests is disturbing... is there a >> high-level bug tracking improving this? It seems like it might be worth >> running a stress test of the tests on a loaded machine to try and shake out >> such problems. >> >> - Daniel >> >>> On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev >>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>> >>>> >>>> On Aug 19, 2016, at 1:42 PM, Enrico Granata <egran...@apple.com >>>> <mailto:egran...@apple.com>> wrote: >>>> >>>>> >>>>> On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev >>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>> >>>>> It looks like this may be the packaging build: >>>>> >>>>>>> >>>>>>> oss-swift-package-linux-ubuntu-15_10 >>>>> >>>>> It was this build: >>>>> >>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7 >>>>> >>>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> >>>>> >>>>> It has since gone blue. >>>>> >>>>> I don’t think I recall ever seeing that test fail before in an >>>>> intermittent way, but that might be what happened here. >>>>> >>>> >>>> Given the failure mode, it does sound likely: >>>> >>>> >>>> File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr >>>> = self._sock.accept() timeout: timed out >>>> >>>> >>>> >>> >>> If we see that again, we may be able to tweak the socket timeout to allow >>> for heavily loaded test CI. >>> >>>>> -Todd >>>>> >>>>>> On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev >>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>> >>>>>> Where is this failure being reported? I’m not seeing anything on >>>>>> swift-3.0-branch or master on ci.swift.org <http://ci.swift.org/>. >>>>>> >>>>>> Kate Stone k8st...@apple.com <mailto:k8st...@apple.com> >>>>>> Xcode Low Level Tools >>>>>> >>>>>>> On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev >>>>>>> <swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org>> wrote: >>>>>>> >>>>>>> + swift-lldb-dev group. >>>>>>> >>>>>>>> On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dun...@apple.com >>>>>>>> <mailto:daniel_dun...@apple.com>> wrote: >>>>>>>> >>>>>>>> Failing in lldb: >>>>>>>> >>>>>>>> Command invoked: >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>> --executable >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>> --rerun-all-issues -C >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>> TestStubReverseConnect.py >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>> --event-add-entries worker_index=14:int Configuration: arch=x86_64 >>>>>>>> compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>> ---------------------------------------------------------------------- >>>>>>>> Collected 2 tests >>>>>>>> ====================================================================== >>>>>>>> ERROR: test_reverse_connect_works_llgs >>>>>>>> (TestStubReverseConnect.TestStubReverseConnect) >>>>>>>> ---------------------------------------------------------------------- >>>>>>>> Traceback (most recent call last): File >>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py", >>>>>>>> line 121, in wrapper func(*args, **kwargs) File >>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>> line 89, in test_reverse_connect_works_llgs >>>>>>>> self.reverse_connect_works() File >>>>>>>> "/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py", >>>>>>>> line 67, in reverse_connect_works (stub_socket, address) = >>>>>>>> self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", >>>>>>>> line 206, in accept sock, addr = self._sock.accept() timeout: timed >>>>>>>> out >>>>>>>> Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9 >>>>>>>> ---------------------------------------------------------------------- >>>>>>>> Ran 2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 >>>>>>>> errors, 1 skipped, 0 expected failures, 0 unexpected successes) >>>>>>>> Session logs for test failures/errors/unexpected successes can be >>>>>>>> found in directory '2016-08-18-15_15_07' >>>>>>>> >>>>>>>> <>[TestStubReverseConnect.py >>>>>>>> FAILED] >>>>>>>> Command invoked: /usr/bin/python >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py >>>>>>>> --executable >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb >>>>>>>> --rerun-all-issues -C >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang >>>>>>>> -s 2016-08-18-15_15_07 --results-port 41187 --inferior -p >>>>>>>> TestStubReverseConnect.py >>>>>>>> /home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test >>>>>>>> --event-add-entries worker_index=14:int >>>>>>>> >>>>>>>> - Daniel >>>>>>>> >>>>>>>>> On Aug 18, 2016, at 9:18 PM, no-re...@swift.org >>>>>>>>> <mailto:no-re...@swift.org> wrote: >>>>>>>>> >>>>>>>>> [FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930] >>>>>>>>> >>>>>>>>> Build URL: >>>>>>>>> https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/ >>>>>>>>> <https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/> >>>>>>>>> Project: oss-swift-package-linux-ubuntu-15_10 >>>>>>>>> Date of build: Thu, 18 Aug 2016 20:26:52 -0700 >>>>>>>>> Build duration: 51 min >>>>>>>>> Identified problems: >>>>>>>>> >>>>>>>>> Regression test failed: This build failed because a regression test >>>>>>>>> in the test suite FAILed. Below is a list of all errors: >>>>>>>>> Indication 1 >>>>>>>>> <https://ci.swift.org//job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7> >>>>>>>>> Changes >>>>>>>>> >>>>>>>>> Commit 2ebab526bfd7a5ce6ef460bebee0399fe669612d by xi_ge: >>>>>>>>> [FixCode] Apply >>>>>>>>> coercion fixits on return statement and initialization >>>>>>>>> >>>>>>>>> edit: test/FixCode/fixits-apply-objc.swift >>>>>>>>> edit: test/FixCode/fixits-apply-objc.swift.result >>>>>>>>> edit: lib/Sema/CSDiag.cpp >>>>>>>>> edit: test/FixCode/fixits-apply.swift.result >>>>>>>>> >>>>>>>>> Commit 6447a2d16edcfbceab940d58a321b3ef70525f1a by daniel_dunbar: >>>>>>>>> [Basic] Update >>>>>>>>> Thread shim to match changes on Linux side. >>>>>>>>> >>>>>>>>> edit: Sources/Basic/Thread.swift >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> swift-lldb-dev mailing list >>>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>>>>> >>>>>> _______________________________________________ >>>>>> swift-lldb-dev mailing list >>>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>>>> >>>>> _______________________________________________ >>>>> swift-lldb-dev mailing list >>>>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>>>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>>>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >>>> >>>> >>>> Thanks, >>>> - Enrico >>>> 📩 egranata@ <about:blank> <about:blank>.com <about:blank> ☎️ 27683 >>> >>> _______________________________________________ >>> swift-lldb-dev mailing list >>> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >>> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >>> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev> >> _______________________________________________ >> swift-lldb-dev mailing list >> swift-lldb-...@swift.org <mailto:swift-lldb-...@swift.org> >> https://lists.swift.org/mailman/listinfo/swift-lldb-dev >> <https://lists.swift.org/mailman/listinfo/swift-lldb-dev>
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
