#2 for me as well On Tue, Feb 25, 2025 at 1:49 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid> wrote:
> #2 seems like the right answer to me as well > > > - ferruzzi > > > ________________________________ > From: Constance Martineau <consta...@astronomer.io.INVALID> > Sent: Tuesday, February 25, 2025 7:45 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [DISCUSS] removing pickling of executor config.... and > any others? > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > +1 for #2. > > #1 for me is a non-starter. Yes, users can set a pod_template_file instead, > but since Airflow will use that file instead of what is set at the > deployment level, users have to know the internals of the cluster to define > it properly, and become responsible for remembering to update things like > the image tag whenever a new image is deployed. The pod override feature > allows you to *override* certain attributes of the pod, like resource > requests and limits, while keeping everything else as is. > > On Tue, Feb 25, 2025 at 10:25 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > #2 for me too. > > > > On Tue, Feb 25, 2025 at 3:23 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > #2 is what I’d go for — that way it’ll “natively” work easily for > things > > > like v1.Pod spec etc which is quite common when using KubeExec — not > > > supporting this would involve a lot of changes to DAGs that I think is > > easy > > > for us to avoid. > > > > > > -ash > > > > > > > On 25 Feb 2025, at 14:18, Daniel Standish > > > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > > > > > Hi we were just reminded of the fact that we still pickle executor > > config > > > > (instead of force it to be json > > > > > > > > (or at least airlfow-json-serializer-serializable) > > > > > > > > Pickling occasionally causes trouble. It did with k8s exec pod > > overrides > > > > when unpickling across airflow versions (and we resolved this by > > running > > > > through airflow serializer). We've seen it with other weird things > > that > > > > users do too. > > > > > > > > There are 3 paths > > > > > > > > 1. force users to use only valid json. so e.g. they would no longer > be > > > able > > > > to supply raw k8s objects as executor config (something that we > > currently > > > > support in pod override) > > > > 2. less extreme, we just run the whole config through the airlfow > json > > > > serializer. this will handle k8s objects, and, various other things, > > but > > > > not everything that users would toss in there > > > > 3. do nothing and leave it so pod override is json encoded first then > > the > > > > whole thing is pickled IIRC > > > > > > > > What say you? > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
