mik-laj commented on pull request #11793: URL: https://github.com/apache/airflow/pull/11793#issuecomment-716940824
> I needed to export GCP_PROJECT_ID, which wasn't written in the doc We are working to make all environmental variables public, but that will take some time. Sorry for the inconvenience. > the GCS_BUCKET variable was used on 2 places, so I changed it to be unique, but only on 1 place, but didn't know whats the problem, so I tried to debug elsewhere, but didn't noticed this :/ (also no env set yet, so I was getting 403 instead of 409) It looks like you don't have access to the right buckets. Each DAG example should allow bucket to be changed using an environment variable. In this particular case, it did not happen, and I think it is worth adding this possibility. This will allow you to use this easy to change buckets in case of a conflicting name. > 2. I'm not really sure what are these tests for. In most cases, we have a more complicated DAG example and the validation is embedded in the DAG logic, e.g. first task creates a resource, and the next task uses this resource. If the first task didn't create it, the next task wouldn't use it. But even without complicated DAGs, such simple system tests without validation can detect problems with third libraries, because when a method with wrong parameters is called, a runtime exception will be thrown. This won't get detected in unit testing as we use unitest.mock very intensively. > Not sure if 1. should be solved as I'm not sure about 2. 😄 But if yes, I can fix both later - gcs_bucket name could be generated and made unique and docs could be updated to mention setting up env vars for system tests specifically. We don't want random names generated every time we run the test as this will make debugging bugs difficult. The bucket name should only be generated once on a given computer to make behavior more predictable. If you want, you can create `files/airflow-breeze-config/init.sh` with some extra environment initialization code. In this directory we store the script that configures the environment variables and provides the service account keys. For now, we have 26 service account key and 128 env variables. This quarter, we've already started working on making these system tests run regularly on the CI. Once we're done, everyone will have an easy way to create all the required environment keys and variables. Running system tests in an OSS project is quite complicated from a variety of financial, security and technical factors. Hope my explanations were helpful for you. Let me know if you want to make any more changes. If not, I will run the system tests (manually) and check if it works properly, then merge the change. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
