potiuk commented on a change in pull request #18494:
URL: https://github.com/apache/airflow/pull/18494#discussion_r716791580
##########
File path: airflow/providers/google/cloud/example_dags/example_cloud_sql.py
##########
@@ -48,8 +49,8 @@
from airflow.utils.dates import days_ago
GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
-INSTANCE_NAME = os.environ.get('GCSQL_MYSQL_INSTANCE_NAME', 'test-mysql')
-INSTANCE_NAME2 = os.environ.get('GCSQL_MYSQL_INSTANCE_NAME2', 'test-mysql2')
+INSTANCE_NAME = os.environ.get('GCSQL_MYSQL_INSTANCE_NAME', 'test-mysql') +
str(random.getrandbits(16))
Review comment:
Yes. This is HIGHLY annoying with cloudsql that name cannot ber reused
for a day or so even if db is deleted.
However I agree random POSTFIX generation in the DAG is a bad idea.
What we used to have in the past is that we had `variables.env` file in
`files/airflow-breeze-config` where we sourced files with variables and we had
a very simple script that generated the random postfix if it was missing.
Then you could do step-by-step testing with keeping the randomly generated
postfix even across breeze restarts.
When you needed to change the database name you'd simply remove the file and
it would be re-generated automatically at breeze entry.
https://github.com/apache/airflow/blob/main/BREEZE.rst#customize-your-environment
Something like that might work (writing it from memory so I am not sure if
it is correct) in variables.env:
```
if [[ ! -f /files/random.env ]]; then
echo "export RANDOM_POSTFIX=${RANDOM}" > /files/random.env
fi
source /files/random.env
export GCSQL_MYSQL_INSTANCE_NAME="test-mysql-${RANDOM_POSTFIX}"
```
This has several nice properties:
* everyone has its own random value
* you keep it stable between runs or even between debug sessions - for
example you could ran tasks from the example DAG separately one-by-one
* you can very easily regenerate the number by simply deleting the
/files/random.env
In the past we even shared the whole `airflow-breeze-config` directory was
actually checked out separate repository where we kept all variables used by
the team. This way you could share different variables between same users who
have access to the same repo - at the same time each of the users will have
different postifx as the random.env would not be part of the repo.
Just an inspiration if you would like to optimize your development workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
