Hi,
Has anyone expremented with making Beam's common element wise and
aggregation transforms more portable? At present a portable runner would
receive the raw proto of a transfrom/ or compiled byte and get executed on
the worker SDK harness.
For example,
Filter transfrom. Filter.greaterThan(3) only a java SDK worker can execute
it because, a non Java runner can only look at it as a raw byes/DoFn form,
and have zero context on what's the filter expression.
But If instead we get the transform and the filter expression as a proto
form.
Example:
```
PTransform {
unique_name: "Filter"
spec {
urn: "beam:transform:filter:v1"
payload {
predicate {
comparison {
op: GREATER_THAN
left {
field { field: "element" }
}
right {
literal { int_value: 3 }
}
}
}
}
}
}
```
The runner can infer the expression and decide on how to execute the
transform instead of delegating execution to worker.
Primary use case for this, we could leverage high performance query engines
like Apache Datafusion to run beam transforms and potentially get db query
level execution speeds. This is kinda of becoming a common pattern, with
projects like Apache Datafusion Comet that leverage datafusion to run Spark
operations. I feel Beam is very well suited for this because of portability
model and we have better infra to port transforms and expressions across
engines and SDKs.
Thanks,
Ganesh.
