addu390 commented on issue #424:
URL: https://github.com/apache/flink-agents/issues/424#issuecomment-3991534748
@xintongsong Here's a slightly more detailed design, along with a draft PR
to show-case the impl.
### Unified Event Design
I'll start with the summary of the goal again,
### Problem
Currently, every custom event requires a dedicated `Java` or `Python` class.
Cross-language events rely on cloudpickle serialization, which is opaque and
Python-specific and cannot define ad-hoc event types without creating new
classes in both languages.
### End Goal
- A single `Event` class that serves both as a base class for subclassed
events and as a direct-use for unified events.
- String-based routing: events route by a `type` string rather than by
Java/Python `class` identity/name.
- JSON-based cross-language Serde: unified events serialize as plain JSON,
no `cloudpickle` dependency for custom events.
- Backward/Full compatibility: existing subclassed events continue to work
unchanged.
### Core Design
Concrete `Event` Class
- The `Event` class now has a `type` (defaults to class-name) field and an
`attributes` map:
- Unified event: new `Event("MyCustomType", Map.of("key", "value"))` sets
`type = "MyCustomType".`
- Subclassed event: `new InputEvent("data")`, falls back to the `class`-name.
- `EventContext` has `eventType` and `eventClass` (new field)
- `eventType`: The routing key. For unified events, this is the
user-defined string (e.g., "`MyCustomType`"). For subclassed events, this is
the fully qualified class name (e.g.,
"`org.apache.flink.agents.api.InputEvent`").
- `eventClass`: Necessary for unified events, this is always
`org.apache.flink.agents.api.Event`. For subclassed events, this matches
`eventType`. `eventClass` tells the deserializer which concrete Java class to
instantiate.
### Cross-Language Flow
For unified events sent from Python:
- User calls `ctx.send_event("MyEvent", key="value").`
- Which constructs `Event(type="MyEvent", attributes={"key": "value"}).`
- The event is serialized to JSON
- The JSON string is passed to `Java` (an existing pattern used in the code
base).
- Java reconstructs the event with `Event.fromJson(json)`, also validates
that the `type` field is present.
- Note: For subclassed events, the existing `cloudpickle` serialization path
is as-is.
### Action Usage
```
@Action(listenEvents = {InputEvent.class})
// class-based
@Action(listenEventTypes = {"MyCustomEvent"})
// string-based
@Action(listenEvents = {InputEvent.class}, listenEventTypes =
{"MyCustomEvent"}) // mixed
```
```
@action(InputEvent) # class-based
@action("MyCustomEvent") # string-based
@action(InputEvent, "MyEvent") # mixed
```
Both resolve to string keys in the `actionsByEvent` map at plan compilation
time:
- Class-based entries use the fully qualified class name (e.g.,
`org.apache.flink.agents.api.InputEvent` in Java).
- String-based entries are passed through as-is (e.g., `MyCustomEvent`).
- At runtime, `event.getType()` produces the routing key, which is looked up
in `actionsByEvent` to find the matching actions.
The design does seem to get into the weeds, not much to really do a deeper
LLD (unless I'm missing something).
As always, put out a draft for the above:
https://github.com/addu390/flink-agents/pull/5/changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
