#2 seems like the right answer to me as well
- ferruzzi ________________________________ From: Constance Martineau <consta...@astronomer.io.INVALID> Sent: Tuesday, February 25, 2025 7:45 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] removing pickling of executor config.... and any others? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. +1 for #2. #1 for me is a non-starter. Yes, users can set a pod_template_file instead, but since Airflow will use that file instead of what is set at the deployment level, users have to know the internals of the cluster to define it properly, and become responsible for remembering to update things like the image tag whenever a new image is deployed. The pod override feature allows you to *override* certain attributes of the pod, like resource requests and limits, while keeping everything else as is. On Tue, Feb 25, 2025 at 10:25 AM Jarek Potiuk <ja...@potiuk.com> wrote: > #2 for me too. > > On Tue, Feb 25, 2025 at 3:23 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > #2 is what I’d go for — that way it’ll “natively” work easily for things > > like v1.Pod spec etc which is quite common when using KubeExec — not > > supporting this would involve a lot of changes to DAGs that I think is > easy > > for us to avoid. > > > > -ash > > > > > On 25 Feb 2025, at 14:18, Daniel Standish > > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > > > Hi we were just reminded of the fact that we still pickle executor > config > > > (instead of force it to be json > > > > > > (or at least airlfow-json-serializer-serializable) > > > > > > Pickling occasionally causes trouble. It did with k8s exec pod > overrides > > > when unpickling across airflow versions (and we resolved this by > running > > > through airflow serializer). We've seen it with other weird things > that > > > users do too. > > > > > > There are 3 paths > > > > > > 1. force users to use only valid json. so e.g. they would no longer be > > able > > > to supply raw k8s objects as executor config (something that we > currently > > > support in pod override) > > > 2. less extreme, we just run the whole config through the airlfow json > > > serializer. this will handle k8s objects, and, various other things, > but > > > not everything that users would toss in there > > > 3. do nothing and leave it so pod override is json encoded first then > the > > > whole thing is pickled IIRC > > > > > > What say you? > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
