@JB: you started working on DOSGi/Karaf runner, no ? couldn't Kuberenetes-based services be similar ?
@Ben: JB is right about the difference between the execution engine (Spark, Flink, Apex) and the resource manager (YARN/Mesos), but you might be able to use Beam directly against services on top of Kuberenetes, though in a very limited way (executing DoFns basically). I would be interested to hear more though, I'm not too familiar with Kubernetes (yet..) but as it becomes popular I'm wondering if/how it could converge with Beam. Amit. On Mon, Nov 21, 2016 at 9:29 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi Ben, > > I did examples with Mesos and Marathon, but not yet with Kubernetes. > Basically, it depends of the execution engines: Beam runner targets an > execution engine (like Spark or Flink). > > So, for instance, as Spark and Flink support Kubernetes, it should not > be a problem (for Spark, regular spark-submit can work with Beam > pipelines with Spark runner). > > Regards > JB > > On 11/21/2016 07:37 PM, Stadin, Benjamin wrote: > > Hi, > > > > Are there any examples for running Beam with Kubernetes? > > > > ~Ben > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
