[ https://issues.apache.org/jira/browse/HADOOP-13716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569723#comment-15569723 ]
Steve Loughran edited comment on HADOOP-13716 at 10/12/16 8:05 PM: ------------------------------------------------------------------- example from a test {code} int reps = eventually( () -> ++count == 4, 50, () -> 10, (timeout, ex) -> new Exception("timeout after " + count) ); {code} This will evaluate the first closure until it succeeds or the total wait time exceeds 50; the operation for interval calculation is always 10. Returns the number of iterations it took. Another example, evaluate a closure until it succeeds. Here it doesn't; the exception caught on the final unsuccessful iteration is rethrown. {code} @Test public void testEvalLambdaFailures() throws Throwable { try { evaluate(() -> { throw new IOException("oops"); }, TIMEOUT, retry); fail("should not have got here"); } catch (IOException expected) { assertExceptionContains("oops", expected); } } {code} As well as being Lambda friendly, the patch is just invoking {{Callable<T>}}, so works with Java & and normal interface/implementation classes; that's how I plan to initially use it for branch-2 code. I just want to be confident we have something which will move to lambda-expressions with ease. Also, I've added a FailFastException, similar to that in S3ATestUtils. If a closure throws that: no attempt to retry. Code that knows what it is doing can use that to bail out fast on a condition which will never be met. I'm looking at doing some S3A work, in particular handling inconsistency problems in some of the FS contract root tests, though I think someone ought to be able to migrate {{waitFor()}} code to it when they want better exception propagation, failure handling or alternative backoff strategies was (Author: ste...@apache.org): example from a test {code} int reps = eventually( () -> ++count == 4, 50, () -> 10, (timeout, ex) -> new Exception("timeout after " + count) ); {code} This will evaluate the first closure until it succeeds or the total wait time exceeds 50; the operation for interval calculation is always 10. Returns the number of iterations it took. Another example, evaluate a closure until it succeeds. Here it doesn't; the e {code} @Test public void testEvalLambdaFailures() throws Throwable { try { evaluate(() -> { throw new IOException("oops"); }, TIMEOUT, retry); fail("should not have got here"); } catch (IOException expected) { assertExceptionContains("oops", expected); } } {code} As well as being Lambda friendly, the patch is just invoking {{Callable<T>}}, so works with Java & and normal interface/implementation classes; that's how I plan to initially use it for branch-2 code. I just want to be confident we have something which will move to lambda-expressions with ease. I'm looking at doing some S3A work, in particular handling inconsistency problems in some of the FS contract root tests, though I think someone ought to be able to migrate {{waitFor()}} code to it when they want better exception propagation, failure handling or alternative backoff strategies > Add an Eventually class to for tests to await eventual completion of lambda > expressions/test clauses > ---------------------------------------------------------------------------------------------------- > > Key: HADOOP-13716 > URL: https://issues.apache.org/jira/browse/HADOOP-13716 > Project: Hadoop Common > Issue Type: New Feature > Components: test > Affects Versions: 2.8.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Attachments: HADOOP-13716-001.patch > > > To make our tests robust against timing problems and eventual consistent > stores, we need to do more spin & wait for state. > We have some code in {{GenericTestUtils.waitFor}} to await a condition being > met, but the predicate it calls doesn't throw exceptions, there's no way for > a probe to throw an exception, and all you get is the eventual "timed out" > message. > We can do better, and in closure-ready languages (scala & scalatest, groovy > and some slider code) we've examples to follow. Some of that work has been > reimplemented slightly in {{S3ATestUtils.eventually}} > I propose adding a class in the test tree, {{Eventually}} to be a > successor/replacement for these. > # has an eventually/waitfor operation taking a predicate that throws an > exception > # has an "evaluate" exception which tries to evaluate an answer until the > operation stops raising an exception. (again, from scalatest) > # plugin backoff strategies (from Scalatest; lets you do exponential as well > as linear) > # option of adding a special handler to generate the failure exception (e.g. > run more detailed diagnostics for the exception text, etc). > # be Java 8 lambda expression friendly > # be testable and tested itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org