It seems like there's a lot of community interest in ML running on Beam -- definitely something that we should eventually have in Beam.
Hopefully, we'll be able to coordinate individual efforts to come up with a unified API. It fits right in with Beam goals to have a library of ML PTransforms that isn't tied to any particular ML backend. Then, users will have portability benefits and will be able to make the right choice for them for each execution. Overall, I think this is a complex feature with a really big impact and benefit to Beam. As such, it would be great to write up and discuss architecture and design in detail first. -- In terms of specific questions, a library of PTransforms would probably be a better start than a DSL (but that doesn't exclude the possibility of a DSL some day). There would be a default implementation, and then each runner could override it, as appropriate. I think Simone's warning should be taken into account, however. Definitely something to have in mind as the design progresses.
