I would go for #2, I do not prefer #1 at all. We do have a well written serde serialiser and in my recent experiment with KE, it could serialise pod overrides in executor_config well.
Thanks & Regards, Amogh Desai On Wed, Feb 26, 2025 at 3:22 AM Vikram Koka <vik...@astronomer.io.invalid> wrote: > #2 for me as well > > On Tue, Feb 25, 2025 at 1:49 PM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> wrote: > > > #2 seems like the right answer to me as well > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Constance Martineau <consta...@astronomer.io.INVALID> > > Sent: Tuesday, February 25, 2025 7:45 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] removing pickling of executor config.... and > > any others? > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > +1 for #2. > > > > #1 for me is a non-starter. Yes, users can set a pod_template_file > instead, > > but since Airflow will use that file instead of what is set at the > > deployment level, users have to know the internals of the cluster to > define > > it properly, and become responsible for remembering to update things like > > the image tag whenever a new image is deployed. The pod override feature > > allows you to *override* certain attributes of the pod, like resource > > requests and limits, while keeping everything else as is. > > > > On Tue, Feb 25, 2025 at 10:25 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > #2 for me too. > > > > > > On Tue, Feb 25, 2025 at 3:23 PM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > > > > > > #2 is what I’d go for — that way it’ll “natively” work easily for > > things > > > > like v1.Pod spec etc which is quite common when using KubeExec — not > > > > supporting this would involve a lot of changes to DAGs that I think > is > > > easy > > > > for us to avoid. > > > > > > > > -ash > > > > > > > > > On 25 Feb 2025, at 14:18, Daniel Standish > > > > <daniel.stand...@astronomer.io.INVALID> wrote: > > > > > > > > > > Hi we were just reminded of the fact that we still pickle executor > > > config > > > > > (instead of force it to be json > > > > > > > > > > (or at least airlfow-json-serializer-serializable) > > > > > > > > > > Pickling occasionally causes trouble. It did with k8s exec pod > > > overrides > > > > > when unpickling across airflow versions (and we resolved this by > > > running > > > > > through airflow serializer). We've seen it with other weird things > > > that > > > > > users do too. > > > > > > > > > > There are 3 paths > > > > > > > > > > 1. force users to use only valid json. so e.g. they would no longer > > be > > > > able > > > > > to supply raw k8s objects as executor config (something that we > > > currently > > > > > support in pod override) > > > > > 2. less extreme, we just run the whole config through the airlfow > > json > > > > > serializer. this will handle k8s objects, and, various other > things, > > > but > > > > > not everything that users would toss in there > > > > > 3. do nothing and leave it so pod override is json encoded first > then > > > the > > > > > whole thing is pickled IIRC > > > > > > > > > > What say you? > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > >
