There are 101 instances of @Ignore, and I've listed them below. A few
takeaways:
- highly concentrated in ZetaSQL, and then second tier in various state
tests specific to a runner
- there are not that many overall, so I'm not sure a report will add much
- they do not all have Jiras
- they do not even all have any explanation at all (some don't leave out
the string parameter, but have an empty string!)
Having a checkstyle that there is a Jira attached seems nice. Then we could
easily grep out the Jiras and not depend on the "sickbay" label.
Raw data (to see the individual items, just do the grep and not the
processing)
% grep --recursive --exclude-dir build '@Ignore' . | cut -d ' ' -f 1 |
sort | uniq -c | sort -r
27
./sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java:
11
./runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java:
7
./runners/spark/src/test/java/org/apache/beam/runners/spark/stateful/SparkStateInternalsTest.java:
7
./runners/apex/src/test/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternalsTest.java:
4
./sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/QueryTest.java:
4
./runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/StructuredStreamingPipelineStateTest.java:
2
./sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlSourceTest.java:
2
./sdks/java/io/mqtt/src/test/java/org/apache/beam/sdk/io/mqtt/MqttIOTest.java:
2
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java:
2
./sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java:
2
./sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/ReduceByKeyTest.java:
2
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java:
2
./sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PCollectionCustomCoderTest.java:
2
./runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java:
1
./sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/sources/UnboundedEventSourceTest.java:
1
./sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5Test.java:
1
./sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduIOTest.java:
1
./sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java:
1
./sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemTest.java:
1
./sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOTest.java:
1
./sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java:@Ignore
("[BEAM-7794]
1
./sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java:@Ignore
("[BEAM-7794]
1
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java:@Ignore
("https://issues.apache.org/jira/projects/BEAM/issues/BEAM-7523";)
1
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java:
1
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java:
1
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java:
1
./sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java:
1
./sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/JoinTest.java:
1
./sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/docs/DocumentationExamplesTest.java:
1
./sdks/java/core/src/test/java/org/apache/beam/sdk/values/PDoneTest.java:
1
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WatchTest.java:
1
./sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java:
1
./sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java:
1
./sdks/java/core/src/test/java/org/apache/beam/sdk/coders/RowCoderTest.java:
1
./runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/SimpleSourceTest.java:
1
./runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/SparkMetricsSinkTest.java:@Ignore
("Has
1
./runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverridesTest.java:
1
./runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java:
1
./runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java:
Kenn
On Tue, May 12, 2020 at 1:24 PM Mikhail Gryzykhin <
gryzykhin.mikh...@gmail.com> wrote:
> I wonder if we can add graph to community metrics showing ignored tests by
> language/project/overall. That can be useful to see focus area.
>
> On Tue, May 12, 2020 at 12:28 PM Jan Lukavský <je...@seznam.cz> wrote:
>
>> +1, visualizing the number of ignored tests in a graph seems useful. Even
>> better with some slices (e.g. per runner, module, ...).
>> On 5/12/20 8:02 PM, Ahmet Altay wrote:
>>
>> +1 to generate a report instead of removing these tests. A report like
>> this could help us with prioritization. It is easier to address issues when
>> we can quantify how much of a problem it is.
>>
>> I am curious what we can do to incentivize reducing the number of
>> flaky/ignored tests? A report itself might provide incentive, it is
>> rewarding to see ignored tests numbers go down over time.
>>
>> On Mon, May 11, 2020 at 8:30 AM Luke Cwik <lc...@google.com> wrote:
>>
>>> Deleting ignored tests does lead us to losing the reason as to why the
>>> test case was around so I would rather keep it around. I think it would be
>>> more valuable to generate a report that goes on the website/wiki showing
>>> stability of the modules (num tests, num passed, num skipped, num failed
>>> (running averages over the past N runs)). We had discussed doing something
>>> like this for ValidatesRunner so we could show which runner supports what
>>> automatically.
>>>
>>> On Mon, May 11, 2020 at 12:53 AM Jan Lukavský <je...@seznam.cz> wrote:
>>>
>>>> I think that we do have Jira issues for ignored test, there should be
>>>> no problem with that. The questionable point is that when test gets
>>>> Ignored, people might consider the problem as "less painful" and postpone
>>>> the correct solution until ... forever. I'd just like to discuss if people
>>>> see this as an issue. If yes, should we do something about that, or if no,
>>>> maybe we can create a rule that test marked as Ignored for long time might
>>>> be deleted, because apparently is only a dead code.
>>>> On 5/6/20 6:30 PM, Kenneth Knowles wrote:
>>>>
>>>> Good point.
>>>>
>>>> The raw numbers are available in the test run output. See
>>>> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Java_Cron/2718/testReport/
>>>> for
>>>> the "skipped" column.
>>>> And you get the same on console or Gradle Scan:
>>>> https://scans.gradle.com/s/ml3jv5xctkrmg/tests?collapse-all
>>>> This would be good to review periodically for obvious trouble spots.
>>>>
>>>> But I think you mean something more detailed. Some report with columns:
>>>> Test Suite, Test Method, Jira, Date Ignored, Most Recent Update
>>>>
>>>> I think we can get most of this from Jira, if we just make sure that
>>>> each ignored test has a Jira and they are all labeled in a consistent way.
>>>> That would be the quickest way to get some result, even though it is not
>>>> perfectly automated and audited.
>>>>
>>>> Kenn
>>>>
>>>> On Tue, May 5, 2020 at 2:41 PM Jan Lukavský <je...@seznam.cz> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> it seems we are accumulating test cases (see discussion in [1]) that
>>>>> are
>>>>> marked as @Ignored (mostly due to flakiness), which is generally
>>>>> undesirable. Associated JIRAs seem to be open for a long time, and
>>>>> this
>>>>> might generally cause that we loose code coverage. Would anyone have
>>>>> idea on how to visualize these Ignored tests better? My first idea
>>>>> would
>>>>> be something similar to "Beam dependency check report", but that seems
>>>>> to be not the best example (which is completely different issue :)).
>>>>>
>>>>> Jan
>>>>>
>>>>> [1] https://github.com/apache/beam/pull/11614
>>>>>
>>>>>
