rsg17 commented on pull request #20769: URL: https://github.com/apache/airflow/pull/20769#issuecomment-1013972440
> > I read through this [guide](https://github.com/apache/airflow/blob/main/TESTING.rst#airflow-system-tests). I think I need make some changes to variables.env file; but I am not sure what these changes should be. > > From your example DAG it looks like you need to set up two variables: > > * GCP_GCS_BUCKET > * CALENDAR_ID > > Apart from that you will need to add a credentials key because you are using it in test: > > ```python > @pytest.mark.credential_file(GCP_GCS_KEY) > ``` > > The file should be named `gcp_gcs.json` and put according to the guide you linked: > > > If your system tests need some credential files to be available for an authentication with external systems, make sure to keep these credentials in the files/airflow-breeze-config/keys directory. Mark your tests with @pytest.mark.credential_file() so that they are skipped if such a credential file is not there. The tests should read the right credentials and authenticate them on their own. The credentials are read in Breeze from the /files directory. The local "files" folder is mounted to the "/files" folder in Breeze. > > Regarding this question: > > > Also, I am not sure about the --forward-credentials option for [running system tests](https://github.com/apache/airflow/blob/main/TESTING.rst#the-typical-system-test-session). Which credentials will it forward? Do I need to put the credentials somewhere? > > You need to use credentials to some GCP project. As far as I know we don't have airflow project that can be used for this purpose (cc @potiuk to confirm). Thank you for all your help @turbaszek. I am running into an authentication error. I don't see this error when I try to connect to gcs directly. Could you help take a look? This gives the error location: `___________________________________ GoogleCalendarToGCSExampleDagsSystemTest.test_run_example_dag_function ____________________________________ /usr/local/lib/python3.7/contextlib.py:73: in inner with self._recreate_cm(): /usr/local/lib/python3.7/contextlib.py:112: in __enter__ return next(self.gen) tests/test_utils/gcp_system_helpers.py:108: in provide_gcp_context f"--key-file={key_file_path}", tests/test_utils/logging_command_executor.py:92: in execute_cmd env=env, /usr/local/lib/python3.7/subprocess.py:800: in __init__ restore_signals, start_new_session) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <subprocess.Popen object at 0x4065c8b250> args = ['gcloud', 'auth', 'activate-service-account', '--key-file=/files/airflow-breeze-config/keys/gcp_gcs.json'], executable = b'gcloud' preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1 p2cwrite = -1, c2pread = 18, c2pwrite = 19, errread = 20, errwrite = 21, restore_signals = True, start_new_session = False ` Here is the error: `E FileNotFoundError: [Errno 2] No such file or directory: 'gcloud': 'gcloud'` I tried running `gcloud auth activate-service-account` with the same key and same service account from my laptop - that worked. I was also able to upload a file to GCS using the same key: https://gist.github.com/rsg17/b582414f630c2ac38323b87213b270d5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
