+1 to ValidatesRunner. I'd be nice if it were (optionally?) parameterized by which feature it validates.
@NeedsRunner is odd, as using a runner is the most natural way to write many (most) tests, but an annotation should be used to mark the exception, not the norm. (I'd just assume a runner is available for all tests, e.g. CoreTests depends on DirectRunner depends on Core). On Thu, Nov 10, 2016 at 10:14 AM, Mark Liu <mark...@google.com.invalid> wrote: > +1 ValidatesRunner > > On Thu, Nov 10, 2016 at 8:40 AM, Kenneth Knowles <k...@google.com.invalid> > wrote: > >> Nice. I like ValidatesRunner. >> >> On Nov 10, 2016 03:39, "Amit Sela" <amitsel...@gmail.com> wrote: >> >> > How about @ValidatesRunner ? >> > Seems to complement @NeedsRunner as well. >> > >> > On Thu, Nov 10, 2016 at 9:47 AM Aljoscha Krettek <aljos...@apache.org> >> > wrote: >> > >> > > +1 >> > > >> > > What I would really like to see is automatic derivation of the >> capability >> > > matrix from an extended Runner Test Suite. (As outlined in Thomas' >> doc). >> > > >> > > On Wed, 9 Nov 2016 at 21:42 Kenneth Knowles <k...@google.com.invalid> >> > > wrote: >> > > >> > > > Huge +1 to this. >> > > > >> > > > The two categories I care most about are: >> > > > >> > > > 1. Tests that need a runner, but are testing the other "thing under >> > > test"; >> > > > today this is NeedsRunner. >> > > > 2. Tests that are intended to test a runner; today this is >> > > > RunnableOnService. >> > > > >> > > > Actually the lines are not necessary clear between them, but I think >> we >> > > can >> > > > make good choices, like we already do. >> > > > >> > > > The idea of two categories with a common superclass actually has a >> > > pitfall: >> > > > what if a test is put in the superclass category, when it does not >> > have a >> > > > clear meaning? And also, I don't have any good ideas for names. >> > > > >> > > > So I think just replacing RunnableOnService with RunnerTest to make >> > clear >> > > > that it is there just to test the runner is good. We might also want >> > > > RunnerIntegrationTest extends NeedsRunner to use in the IO modules. >> > > > >> > > > See also Thomas's doc on capability matrix testing* which is aimed at >> > > case >> > > > 2. Those tests should all have a category from the doc, or a new one >> > > added. >> > > > >> > > > * >> > > > >> > > > >> > > https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2 >> > VlUyVtpi2WzzGM/edit >> > > > >> > > > Kenn >> > > > >> > > > On Wed, Nov 9, 2016 at 12:20 PM, Jean-Baptiste Onofré < >> j...@nanthrax.net >> > > >> > > > wrote: >> > > > >> > > > > Hi Mark, >> > > > > >> > > > > Generally speaking, I agree. >> > > > > >> > > > > As RunnableOnService extends NeedsRunner, @TestsWithRunner or >> > > > @RunOnRunner >> > > > > sound clearer. >> > > > > >> > > > > Regards >> > > > > JB >> > > > > >> > > > > >> > > > > On 11/09/2016 09:00 PM, Mark Liu wrote: >> > > > > >> > > > >> Hi all, >> > > > >> >> > > > >> I'm working on building RunnableOnService in Python SDK. After >> > having >> > > > >> discussions with folks, "RunnableOnService" looks like not a very >> > > > >> intuitive >> > > > >> name for those unit tests that require runners and build >> lightweight >> > > > >> pipelines to test specific components. Especially, they don't have >> > to >> > > > run >> > > > >> on a service. >> > > > >> >> > > > >> So I want to raise this idea to the community and see if anyone >> have >> > > > >> similar thoughts. Maybe we can come up with a name this is tight >> to >> > > > >> runner. >> > > > >> Currently, I have two names in my head: >> > > > >> >> > > > >> - TestsWithRunners >> > > > >> - RunnerExecutable >> > > > >> >> > > > >> Any thoughts? >> > > > >> >> > > > >> Thanks, >> > > > >> Mark >> > > > >> >> > > > >> >> > > > > -- >> > > > > Jean-Baptiste Onofré >> > > > > jbono...@apache.org >> > > > > http://blog.nanthrax.net >> > > > > Talend - http://www.talend.com >> > > > > >> > > > >> > > >> > >>