potiuk commented on PR #31565:
URL: https://github.com/apache/airflow/pull/31565#issuecomment-1571628042

   
   > 1. On the implementation side I have a hard time following what you are 
describing. The pydantic models in `serialization` are straight copies from the 
ORM counterparts. Are you referring to future implementations?
   
   Yes. The idea is to add methods that could be called when the Pydantic model 
is used on the client side. The assumption here is that we want to have non-ORM 
models. that will behave same ways as the DB models when we call their methods 
and those (public) methods should implement the same behaviour as their ORM 
counter-parts.
   
   > 2. I don't think `pydantic-sqlalchemy` as is was suitable for Airflow, but 
I think it could be extended or re-used so that it would work for us. For 
example, the `relationship` challenge could probably easily be solved by using 
a proxy and can be detected automatically.
   
   It would be rather complex and brittle as I see. Especially that some of 
those methods we are going to call will be straight using fields we already 
have in the pydantic class and some will have to make a remote call. I tried 
dynamically generatic "non-ORM" classes that would do that, but that gets 
complex very quickly, and so far my assesment is that explicit copying of the 
method invocation in the Pydantic version of the class is simply simpler. And 
due to the public nature and small-ish number of those - except having to 
extract common code to call an having to proxy the call to those common 
implementations, the maintenance overhead is very low. We aren't going to 
change the signatures of those methods (they are public same as fields) and the 
duplication will limit to declaration of the fields and methods. 
   
   The assumption is here that we should **not** create the SQLAlchemy DB 
models when we have no database (this will happen in the client mode). Instead 
the serialization will return a Pydantic version of those and we need to bring 
mostly the same interface there.
   
   So from my earlier trials, being explicit and duplicating those is the 
simplest and most straightforward ways of doing it.
   
   > 3. On your [setup.py download URL 
#4](https://github.com/apache/airflow/issues/4) to me this seems a quite common 
thing that happens in RPC context. The 
`context.ti.get_num_running_task_instances()` just calls the Rest API instead 
of the ORM API to obtain what is required. Maybe I am looking at this with a 
too simple view?
   
   Yes - in many cases it will be lilke that, but in some other cases we will 
just use the fields we already have. A number of those methods do not require 
the RPC call at all, they can just provide the same result by passing the 
TaskInstance | TaskInstancePydantic object to it. Simple example: 
`email_alert(self, exception, task: BaseOperator)` will not need to do RPC call 
to send email. So while we could potentially make "any" call to the 
TaskPydantic method execute a remote call, the idea is to do it only when 
necessary.
   
   > 4. I would have loved that we took the chance when moving to a Managed API 
that we would drop certain things that really shouldn't be available. So slowly 
moving away from the ORM model to the API model.
   
   Me too. I would absolutely love it. But that - unfortunately means heavily 
braking changes for those few models that we decided to expose I am afraid.
   
   > 5. In [Add Pydantic-powered ORM models serialization for internal API. 
#29776](https://github.com/apache/airflow/pull/29776) I would have preferred 
not using the `json` method from pydantic, but to use the `dict` method. Which 
allows us freedom of choice down the line and use a different encoder (JSON or 
whatever) in the future. Basically, _any_ serialization should go through 
`serde` in my opinion as it does the right thing (TM ;-), like versioning and 
including class information) and if it doesn't there is a centralized place 
where we can fix it. The choice for pydantic in [Add Pydantic-powered ORM 
models serialization for internal API. 
#29776](https://github.com/apache/airflow/pull/29776) was due to `@dataclass` 
or `@attr` not serializing classes en related entities. But that is exactly 
what `serde` does for you. The only advantage pydantic currently has, is that 
it does this recursively and it keeps the structure which `@attr` and 
`@dataclass` don't do (that is a limitatio
 n of those classes not of `serde`). However, the current implementation of the 
ORMless models does not require that.
   
   Yes. From serialization point of view this is fine - we can send the model's 
data. And this is fine, I have no problem with how it is implemented now, it's 
just we need to somehow tell the serialized object on the client side to expose 
most of the same methods that the ORM objects expose, and to the right proxying 
in those cases. For now I think explicitly having *Pydantic object and 
explicitly implemented methods that will do the proxying will be far easier to 
maintain than trying to take the ORM model and dynamically generate a 
corresponding class on the client side that will do the proxying.
   
   I believe if we try to do it automatically, we can have complex, dynamic 
code to maintain that will be difficult to understand and follow. Compare it 
with "straight" proxying the methods with some signature and field duplication. 
I believe the latter is much simpler to maintain in the long term, especially 
when helped with MyPy static checks and Task | TaskPydantic union used in 
places where both can be used.
   
   > 6. Your mypy reference makes sense. But can't this be solved as well with 
inheritance? E.g. have a ORM implementation and API implementation that both 
inherit from an common ancestor?
   
   I think this is a bit more pythonic way to do the inheritance - if we have 
common ancestor to SqlAlchemy model - the duplication goes there. You will have 
to have ORM fields defined in the model and same fields declared in the 
"ancestor". IMHO the TaskInstance | TaskInstancePydantic is a form of that. We 
can even introduce a new Type  "SerializableTaskInstance = TaskInstance| 
TaskInstancePydantic` and use it instead. You also mentiond In many cases when 
TaskInstance is used (in core) there is absolutlely no need to use TaskInstance 
| TaskInstacePydantic. This will only be needed for methods that LocalJobRunner 
uses them (so in case when the client potentially has ORM/non-ORM variant). 
Which is also a nice indication that this code can run in both contexts.
   
   I'd love to do it differently as well but I do not know how.
   
   > 7. In case of code duplication, why not use generation at compile time 
instead of runtime if the above cannot be done? From the ORM models we can 
define what is required and generate the pydantic models at compile time. BTW: 
the way pydantic models seem to be used now are closely related to what 
protobuf does imho.
   
   > What it boils down to is that I do think that your 3 and 4 can be solved 
by using (generated) proxies that call a Rest API. I would be surprised if you 
didnt look at that yet so what am I missing?
   
   Yes. I thought (and tried) that but I could not see a simple way of doing 
it. My attempts to do so end up in a way that will be difficult to follow by 
anyone, as opposed to having the two types of objects (TaskInstance ORM, 
TaskInstacePydantic - non-ORM) explicitly defined (even with some duplication 
and manual proxying of existing public methods). 
   
   But yes, maybe I am wrong, maybe it's a misjudgmenet. I would love to see a 
POC, simple variant of that to compare if we think it can be done simpler. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to