Hi Kenneth, Try using the RDD.pipe() operator in Spark, which lets you call out to an external process by passing data to it through standard in/out. This will let you call programs written in C# (e.g. that use your ML libraries) from a Spark program.
I believe there are other projects enabling communication from Java to .NET, e.g. http://jni4net.sourceforge.net, but I’m not sure how easy they’ll be to use. Matei On Dec 16, 2013, at 10:54 AM, Kenneth Tran <[email protected]> wrote: > Hi, > > We have a large ML code base in .NET. Spark seems cool and we want to > leverage it. What would be the best strategies to bridge the our .NET code > and Spark? > > Initiate a Spark .NET project > A lightweight bridge between .NET and Java > While (1) sound too daunting, it's not clear to me how to do (2) easily and > efficiently. > > I'm willing to contribute to (1) if there's already an existing effort. >
