This is a long-standing JUnit bug, and I suspect these tests have never run.
I've tested JUnit 4.13-beta1, and it partially fixes the issue.
Parameterized tests now work as long as the include category is set on the
entire test class. However individual methods marked with the category are
still not executed. So, this works:
@RunWith(Parameterized.class)
@Category(NeedsRunner.class)
public static class NeedsRunnerTests {
@Test
public void testFOO() throws Throwable {
}
}
But the following does not work:
@RunWith(Parameterized.class)
public static class NeedsRunnerTests {
@Test
@Category(NeedsRunner.class)
public void testWriteThenReadJavaClass() throws Throwable {
}
}
Since no more JUnit 4 releases are planned (and last I checked we had
issues moving to JUnit 5), we might need to refactor
the parameterized tests so that that the NeedsRunner (or ValidatesRunner)
tests are in their own test class.
Reuven
On Tue, Dec 18, 2018 at 10:45 AM Kenneth Knowles <[email protected]> wrote:
> Upon digging in, it appears to be a JUnit issue. Did it ever work? The
> includes/excludes are passed along from Gradle to JUnit. Fixed in the
> as-yet-still-beta 4.13 (notably in RC stages since Feb, and declared to be
> the last release of JUnit 4).
>
> Kenn
>
> On Tue, Dec 18, 2018 at 12:45 PM Scott Wegner <[email protected]> wrote:
>
>> We hit a similar issue when trying to refactor some tests to use the
>> 'Enclosed' runner. It seems the Gradle test filtering is not compatible
>> with the Enclosed runner See BEAM-4375 [1] and pr/5440 [2].
>>
>> I'm not familiar with the Parametrized runner, but perhaps it has similar
>> issues.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-4375
>> [2] https://github.com/apache/beam/pull/5440
>>
>> On Mon, Dec 17, 2018 at 6:15 PM Kenneth Knowles <[email protected]> wrote:
>>
>>> Also FWIW when I tried to repro this, I consistently got many failures
>>> and a hang when I tried to run the whole suite via `./gradlew
>>> needsRunnerTests`. Filed https://issues.apache.org/jira/browse/BEAM-6256
>>> with some details.
>>>
>>> Kenn
>>>
>>> On Mon, Dec 17, 2018 at 8:54 PM Reuven Lax <[email protected]> wrote:
>>>
>>>> It looks like maybe this is a JUnit bug:
>>>> https://github.com/gradle/gradle/issues/1153
>>>>
>>>> The fix is targeted for JUnit 4.13, which is still not fully released
>>>> (and is intended to be the final JUnit 4 release). 4.13.-beta1 is out
>>>> though, and it might be worth trying that out to see if it fixes this. Does
>>>> anyone know how we control the version of JUnit used?
>>>>
>>>> On Mon, Dec 17, 2018 at 5:39 PM Reuven Lax <[email protected]> wrote:
>>>>
>>>>> Ok, this is worrying - we're silently skipping some important test
>>>>> suites. Does anyone know what might be causing us to skip them?
>>>>>
>>>>> On Mon, Dec 17, 2018 at 5:03 PM Kenneth Knowles <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Flipping through the latest Java PreCommit Cron, it does look like
>>>>>> NeedsRunner tests within the Parameterized sub-suites of TextIOReadTest
>>>>>> are
>>>>>> not executed.
>>>>>>
>>>>>> https://scans.gradle.com/s/7s23sf6cgmniw/tests
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Sat, Dec 15, 2018 at 12:16 PM Reuven Lax <[email protected]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I tried to switch one of our tests (AvroIOTest) over to be a
>>>>>>> parameterized test. However after doing so, I cannot get the NeedsRunner
>>>>>>> tests to execute! All attempts fail with the following:
>>>>>>>
>>>>>>> No tests found for given includes:
>>>>>>> [org.apache.beam.sdk.io.AvroIOTest](filter.includeTestsMatching)
>>>>>>>
>>>>>>> I also can't find evidence that Jenkins will execute this either.
>>>>>>>
>>>>>>> Am I missing something, or do our NeedsRunner tests no longer work
>>>>>>> for parameterized tests? If the latter this is concerning, as there are
>>>>>>> a
>>>>>>> number of NeedsRunner paramaterized tests in our codebase (e.g. the
>>>>>>> TextIO
>>>>>>> tests).
>>>>>>>
>>>>>>> Reuven
>>>>>>>
>>>>>>
>>
>> --
>>
>>
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>
