potiuk commented on PR #30175: URL: https://github.com/apache/airflow/pull/30175#issuecomment-1489923632
FYI: I just started discussion on our devlist about formalising policies about approving new providers https://lists.apache.org/thread/xf4wn1hw08cs725fp962252y7mmptk3b My proposal is to be pretty receptive to providers like Kafka (I think we need it). One question there which I have a preference but I will see what others think is to expect from providers like Kafka to also have integration tests implemented (we already have them for mongo, pinot. trino, cassandra, celery and kerberos - so it seems appropriate to also have them for Kafka). @dylanbstorey -> what do you think about adding integration tests for the Kafka provider. In short: * they are described here: https://github.com/apache/airflow/blob/main/TESTING.rst#airflow-integration-tests * basically what the tests are about is that you use a docker image with the "software" being tested (kafka in this case) and integrate with Breezes "docker compose" framework in the way to allow breeze to start such an image via `--integration` switch * they have a dedicated "integration" (pytest driven) test cases that instead of mocking the software in question perform a few basic operations with the "actually running" software * they have a separate job in our CI - for example https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089 - where the integrations are started (via breeze) and the tests are run The nice thing about it is that it also makes it easier for the author to be sure that things work as expected - with real services, and when there are regressions, it makes it much easier for anyone to start such integration and fix problems, without having to set up everything from scratch (also it allows to test if the integration works with newever version of Kafka for example - usually by just changing a version of the integration docker image used and making a PR). Ultimately, this is a way also for the author to be able to perform a hand-over of such provider to the community and make it easier for others to step-in and fix problems. I am happy to help with implementing those tests. So @dylanbstorey - WDYT? Would that be too much to ask? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
