> On Dec. 11, 2014, 1:39 a.m., Chris Mattmann wrote: > > > > Michael Starch wrote: > I put all of the Spark stuff in a spark package so that it is seperated > from the rest of the resource manager. It is an inter-dependent set of code. > Do you really want me to sort its components out in to the various other > subdirectories? If so, I can.
Thanks Mike, yes I understand that. And yes, in fact, I'd like you to sort them into the various sub directories b/c that is the architecture of the resource manager. It shouldn't matter that this is Spark, etc. We have an XML-RPC and Ganglia, etc., set of plugins that are part of the architectural sub folders for the resource manager. We shouldn't introduce new top level directories and folders unless we are actually creating new extension points, which we are not here. We are simply implementing plugins that rely on Spark and Mesos, and those plugins should fit in the extension point folders for Resource Manager as they are currently defined. - Chris ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28917/#review64662 ----------------------------------------------------------- On Dec. 10, 2014, 9:38 p.m., Michael Starch wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28917/ > ----------------------------------------------------------- > > (Updated Dec. 10, 2014, 9:38 p.m.) > > > Review request for oodt and Chris Mattmann. > > > Repository: oodt > > > Description > ------- > > This change allows Spark jobs to be run in the resource manager. It > integrates with a generic Spark daemon, so spark can be run locally, as a > test spark cluster, and in mesos (not affiliated with the resouce manager > mesos integration). > > This is a solution to: https://issues.apache.org/jira/browse/OODT-780 > > > Diffs > ----- > > trunk/core/pom.xml 1644524 > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java > PRE-CREATION > > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java > PRE-CREATION > trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml > PRE-CREATION > trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION > trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml > PRE-CREATION > trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml > PRE-CREATION > trunk/resource/src/main/resources/resource.properties 1644524 > > trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala > PRE-CREATION > > Diff: https://reviews.apache.org/r/28917/diff/ > > > Testing > ------- > > Demonstration created, demonstration jobs run and benchmarks taken. > > > Thanks, > > Michael Starch > >
