potiuk commented on PR #27854: URL: https://github.com/apache/airflow/pull/27854#issuecomment-1325568521
> I was actually thinking of system tests as you mentioned because we need here to test "in real" the operators. But you also have very valid points regarding the diversity of databases/versions and also the complexity of setting up these tests Just for reference of the "Android OS" that I wrote above - this is the best approach I've ever seen working well for very similar case (I've been developing custom Android OS and used it on a daily basis: Here are some of the relevant files from it: Description of the process (a bit) https://cs.android.com/android/platform/superproject/+/master:prebuilts/sdk/current/placeholder-api.txt Current Android API: https://cs.android.com/android/platform/superproject/+/master:prebuilts/sdk/current/androidx-api.txt Past android APIs (there are many) - this is 21: https://cs.android.com/android/platform/superproject/+/master:prebuilts/sdk/21/public/api/android.txt Script to update the API: https://cs.android.com/android/platform/superproject/+/master:prebuilts/sdk/current/update_androidx-api.sh The build process is such, that whenever any of those "public" API changes, your build fails - until your run the script and commit the change by confirming you are OK with that change and that you realize what consequences are. This makes it super straightforward what "public" API is, and what was the state of it at any given time. And there are various tests checking if only those public API methods are used. It's FAR easier in Java - because of encapsulation and "real private" - but I think we should be able to come up with something similar. The big benefits of such approach are, that: a) you know exactly what "public" API is b) you fail the build if you accidentally modify it c) the user who commits the change knows that it's the user's responsibility to review if the change has "breaking characteristics" also for past versions of relased code d) reviewer knows that things has to be reviewed REALLY THOROUGHLY when API changes in any way (and you see exactly what changed vs. past released versions) I think something similar might be good for us (and I will add some pre-commit for that shortly). This has a very low "investment" effort and while you do not run comprehensive suite of "whether it works" tests, you have at least a chance and opportunity for both - developer and tester to be flagged "this is dangerous, make sure you know what you are doing". I think this is the best way of applying automation for such cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
