GitHub user tdas opened a pull request:

    [SQL][STREAMING][TEST] Fix flaky tests in StreamingQueryListenerSuite

    This work has largely been done by @lw-lin in his PR #15497. This is a 
slightly refactoring of it.
    ## What changes were proposed in this pull request?
    There were two sources of flakiness in StreamingQueryListener test.
    - When testing with manual clock, consecutive attempts to advance the clock 
can occur without the stream execution thread being unblocked and doing some 
work between the two attempts. Hence the following can happen with the current 
    |      StreamExecution thread       |         testing thread         |
    |  ManualClock.waitTillTime(100) {  |                                |
    |        _isWaiting = true          |                                |
    |            wait(10)               |                                |
    |        still in wait(10)          |  if (_isWaiting) advance(100)  |
    |        still in wait(10)          |  if (_isWaiting) advance(200)  | <- 
this should be disallowed !
    |        still in wait(10)          |  if (_isWaiting) advance(300)  | <- 
this should be disallowed !
    |      wake up from wait(10)        |                                |
    |       current time is 600         |                                |
    |       _isWaiting = false          |                                |
    |  }                                |                                |
    - Second source of flakiness is that the adding data to memory stream may 
get processing in any trigger, not just the first trigger.
    My fix is to make the manual clock wait for the other stream execution 
thread to start waiting for the clock at the right wait start time. That is, 
`advance(300)` (see above) will wait for stream execution thread to complete 
the wait that started at time 100, and start a new wait at time 300 (i.e. time 
stamp after the previous `advance(200)`).
    In addition, since this is a feature that is solely used by 
StreamExecution, I removed all the non-generic code from ManualClock and put 
them in StreamManualClock inside StreamTest.
    ## How was this patch tested?
    Ran existing unit test MANY TIME in Jenkins

You can merge this pull request into a Git repository by running:

    $ git pull metrics-flaky-test-fix

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15519
commit 5bc47b639ede049f44ad4f47a88d26219fea6193
Author: Liwei Lin <>
Date:   2016-10-15T02:21:58Z

    Fix flaky test

commit eb59a98146f30163675cec3b52f69fedd7a234fc
Author: Liwei Lin <>
Date:   2016-10-17T13:15:40Z

    Revert "Fix flaky test"
    This reverts commit 5bc47b639ede049f44ad4f47a88d26219fea6193.

commit 7ae7782cdede0c3f2a3db0a09401cf0d682a264f
Author: Liwei Lin <>
Date:   2016-10-17T11:53:46Z

    Fix flaky test again

commit 6fdbae34a6e806ad0ca8bb6cfd6ff630e0b84143
Author: Tathagata Das <>
Date:   2016-10-17T20:46:48Z

    Refactored Manual clock


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to