vincbeck commented on PR #30003: URL: https://github.com/apache/airflow/pull/30003#issuecomment-1476841581
> > I'm with Elad on this. Having just a couple of examples left in examples folder is rather confusing for user's pov. Providing all examples / system tests in one place makes it easier to reference other tests as examples when writing a DAG. > > > Are people going to be confused why we don't run many of these tests? So far we only have migrated examples-->tests that we are committed to running in our test setup. We won't be running many of the tests in this PR since we don't have the external accounts and resources required to run them (many of the transfers are a good example of this). > > Yes it's a valid worry. But maybe we could make it clearer by slight modification of the pytest code so that it will raise Skip exception for those tests unless we setup some variable (RUN_EXPERIMENTAL_TESTS and providing skip reason). This will be visible in the code, as well as in your dashboard you could run whole folder, and possibly even report number of skipped tests. I think this both removes the confusion and makes it clear which tests are "tested". My concern with this approach is we would end up having 2 categories of system tests: experimental and non experimental ones. IMHO, we should not have such categorization, users are welcome to run them if they want regardless of whether AWS run them as part of their health dashboard. Moreover, forcing users to have the flag `RUN_EXPERIMENTAL_TESTS` on to run these tests would make it more difficult for users to run them (or would incentivize them to NOT run them). We made some recent changes in the health dashboard so that system tests which are not run are listed in the dashboard but with no information (since they are not run). [See dashboard](https://aws-mwaa.github.io/open-source/system-tests/dashboard.html). If we go with this change, then all example DAGs would show up in the dashboard and styled like `example_dms` or `example_emr_notebook_execution`. To me this makes sense. We can go further and move system tests which are not run to the bottom of the dashboard. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
