pltbkd commented on issue #1016: URL: https://github.com/apache/flink-agents/issues/1016#issuecomment-5621646337
Yes — this problem also surfaced in the sub-agent framework work, and it was discussed in #926. The durable id mechanisms of Java and Python differ, and Python's original mechanism cannot distinguish multiple lambdas captured by the same method. Given the different iteration needs on each side, and to keep coordination cost low, we did not previously force the two onto the same scheme. But now both PRs are merged, and the pre-0.4 API cleanup is coming, so I think this is a good moment to address it. In an earlier community meeting we reached a consensus: Java and Python differ in language characteristics — since Java methods are not first-class citizens, some natural Python usages require complex interface implementations in Java. We should respect Python's ergonomics, not force Python into a strict correspondence with Java, and instead keep the two sides semantically consistent. In the sub-agent framework PR, `durable_id` was a hard requirement. We introduced `durable_id` as an optional parameter of the `durable_execute` family, and to avoid parameter contagion we introduced the `with_durable_id` / `get_durable_id` helpers, which inject the `durable_id` into the durable call instead of actually constructing a type. This lets users opt into a deterministic `durable_id`, while the default derivation is still used when none is set explicitly. Users keep their Python usage habits and upgrade compatibility is unaffected, but when there is a need they can specify a deterministic id to avoid the lambda-capture problem. In the durable batch execution PR, the `DurableCall` class was introduced to encapsulate a single call within a batch, but to stay aligned with the existing mechanism it did not add any `durable_id`-related field. I understand this is also the background of the current problem. There are possibly several improvement directions, with corresponding divergences: 1. Should `DurableCall` gain a `durable_id` field, or should the `func` passed into `DurableCall` be wrapped with `with_durable_id`? 2. If `DurableCall` gains a `durable_id` field, should `durable_execute` accept a `DurableCall`? Further, should `with_durable_id` then be removed, unifying this need onto `DurableCall` with `durable_id`? 3. Should the automatic digest derivation be kept? It favors Python's ergonomics and avoids upgrade risk for existing jobs, but may implicitly bring the lambda information-loss problem. My personal preference is the `with_durable_id` approach: construct `DurableCall` with a `func` wrapped by `with_durable_id`. On the digest question, my leaning is: when a deterministic id is used, recovery no longer matches the digest; when no id is provided, deriving the digest is still allowed as the fallback. I think this is not only a design question of the durable execute mechanism itself, but also a matter of API tidiness. We still have the chance to change the API form, so we can shape the API into its most suitable form. What do you all think would be best? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
