potiuk commented on PR #31565: URL: https://github.com/apache/airflow/pull/31565#issuecomment-1571628042
> 1. On the implementation side I have a hard time following what you are describing. The pydantic models in `serialization` are straight copies from the ORM counterparts. Are you referring to future implementations? Yes. The idea is to add methods that could be called when the Pydantic model is used on the client side. The assumption here is that we want to have non-ORM models. that will behave same ways as the DB models when we call their methods and those (public) methods should implement the same behaviour as their ORM counter-parts. > 2. I don't think `pydantic-sqlalchemy` as is was suitable for Airflow, but I think it could be extended or re-used so that it would work for us. For example, the `relationship` challenge could probably easily be solved by using a proxy and can be detected automatically. It would be rather complex and brittle as I see. Especially that some of those methods we are going to call will be straight using fields we already have in the pydantic class and some will have to make a remote call. I tried dynamically generatic "non-ORM" classes that would do that, but that gets complex very quickly, and so far my assesment is that explicit copying of the method invocation in the Pydantic version of the class is simply simpler. And due to the public nature and small-ish number of those - except having to extract common code to call an having to proxy the call to those common implementations, the maintenance overhead is very low. We aren't going to change the signatures of those methods (they are public same as fields) and the duplication will limit to declaration of the fields and methods. The assumption is here that we should **not** create the SQLAlchemy DB models when we have no database (this will happen in the client mode). Instead the serialization will return a Pydantic version of those and we need to bring mostly the same interface there. So from my earlier trials, being explicit and duplicating those is the simplest and most straightforward ways of doing it. > 3. On your [setup.py download URL #4](https://github.com/apache/airflow/issues/4) to me this seems a quite common thing that happens in RPC context. The `context.ti.get_num_running_task_instances()` just calls the Rest API instead of the ORM API to obtain what is required. Maybe I am looking at this with a too simple view? Yes - in many cases it will be lilke that, but in some other cases we will just use the fields we already have. A number of those methods do not require the RPC call at all, they can just provide the same result by passing the TaskInstance | TaskInstancePydantic object to it. Simple example: `email_alert(self, exception, task: BaseOperator)` will not need to do RPC call to send email. So while we could potentially make "any" call to the TaskPydantic method execute a remote call, the idea is to do it only when necessary. > 4. I would have loved that we took the chance when moving to a Managed API that we would drop certain things that really shouldn't be available. So slowly moving away from the ORM model to the API model. Me too. I would absolutely love it. But that - unfortunately means heavily braking changes for those few models that we decided to expose I am afraid. > 5. In [Add Pydantic-powered ORM models serialization for internal API. #29776](https://github.com/apache/airflow/pull/29776) I would have preferred not using the `json` method from pydantic, but to use the `dict` method. Which allows us freedom of choice down the line and use a different encoder (JSON or whatever) in the future. Basically, _any_ serialization should go through `serde` in my opinion as it does the right thing (TM ;-), like versioning and including class information) and if it doesn't there is a centralized place where we can fix it. The choice for pydantic in [Add Pydantic-powered ORM models serialization for internal API. #29776](https://github.com/apache/airflow/pull/29776) was due to `@dataclass` or `@attr` not serializing classes en related entities. But that is exactly what `serde` does for you. The only advantage pydantic currently has, is that it does this recursively and it keeps the structure which `@attr` and `@dataclass` don't do (that is a limitatio n of those classes not of `serde`). However, the current implementation of the ORMless models does not require that. Yes. From serialization point of view this is fine - we can send the model's data. And this is fine, I have no problem with how it is implemented now, it's just we need to somehow tell the serialized object on the client side to expose most of the same methods that the ORM objects expose, and to the right proxying in those cases. For now I think explicitly having *Pydantic object and explicitly implemented methods that will do the proxying will be far easier to maintain than trying to take the ORM model and dynamically generate a corresponding class on the client side that will do the proxying. I believe if we try to do it automatically, we can have complex, dynamic code to maintain that will be difficult to understand and follow. Compare it with "straight" proxying the methods with some signature and field duplication. I believe the latter is much simpler to maintain in the long term, especially when helped with MyPy static checks and Task | TaskPydantic union used in places where both can be used. > 6. Your mypy reference makes sense. But can't this be solved as well with inheritance? E.g. have a ORM implementation and API implementation that both inherit from an common ancestor? I think this is a bit more pythonic way to do the inheritance - if we have common ancestor to SqlAlchemy model - the duplication goes there. You will have to have ORM fields defined in the model and same fields declared in the "ancestor". IMHO the TaskInstance | TaskInstancePydantic is a form of that. We can even introduce a new Type "SerializableTaskInstance = TaskInstance| TaskInstancePydantic` and use it instead. You also mentiond In many cases when TaskInstance is used (in core) there is absolutlely no need to use TaskInstance | TaskInstacePydantic. This will only be needed for methods that LocalJobRunner uses them (so in case when the client potentially has ORM/non-ORM variant). Which is also a nice indication that this code can run in both contexts. I'd love to do it differently as well but I do not know how. > 7. In case of code duplication, why not use generation at compile time instead of runtime if the above cannot be done? From the ORM models we can define what is required and generate the pydantic models at compile time. BTW: the way pydantic models seem to be used now are closely related to what protobuf does imho. > What it boils down to is that I do think that your 3 and 4 can be solved by using (generated) proxies that call a Rest API. I would be surprised if you didnt look at that yet so what am I missing? Yes. I thought (and tried) that but I could not see a simple way of doing it. My attempts to do so end up in a way that will be difficult to follow by anyone, as opposed to having the two types of objects (TaskInstance ORM, TaskInstacePydantic - non-ORM) explicitly defined (even with some duplication and manual proxying of existing public methods). But yes, maybe I am wrong, maybe it's a misjudgmenet. I would love to see a POC, simple variant of that to compare if we think it can be done simpler. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
