SamWheating commented on issue #28086:
URL: https://github.com/apache/airflow/issues/28086#issuecomment-1535680297
Started seriously looking into this today and I had a few design decisions
that I wanted to get some early feedback on -
#### Basic implementation
I'm thinking that I'd follow the approach suggested above of just creating
secrets in the task namespace at execution time, and then deleting them in the
`cleanup` and `on_kill` methods. Are there any reasons you can think of that
this might be a bad approach?
#### Providing connection-backed secrets to the KubernetePodOperator
I have two potential approaches here:
1) Add a `KubernetesConnectionSecret` class which stores all of the
information required for mounting and provisioning a connection-backed secret,
and then add a `connection_secrets: List[KubernetesConnectionSecret]` arg to
the KubernetesPodOperator.
2) Add a `connection_id` argument to the existing `Secret` class, and if
this is defined then we can consider this to be a connection-backed secret and
handle it appropriately.
I think I am leaning towards (2) since it keeps things simple within the
operator interface, but I'd totally understand if people don't want to overload
the `Secret` class. Thoughts?
#### Naming connection-backed secrets in kubernetes
I believe kubernetes secret names are limited to 253 characters, so creating
something truly unique like `{DAG_ID}-{TASK_ID}-{EXECUTION_DATE}-{TRY_NUMBER}`
would almost definitely run into length issues.
I think that it might just be easiest to call it something like
`{DAG_ID[:64]}-{TASK_ID[:64]}-{RANDOM_16_CHAR_SHA}`. This would avoid
collisions, but also remove any identifying information from the kube
resources, making things like re-use between tries and also potentially
creating a mess for kube cluster admins.
Anyways, I will start working on an early implementation, but please let me
know what you think of these proposals as there's a lot of room for
interpretation here 😄
cc @potiuk @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
