hdfs 1.0.4 but we primarily use Cassandra + Spark (calliope). I tested it with both Are you using it with HDFS? What version of Hadoop? 1.0.4? Ognen
On 3/10/14, 8:49 PM, abhinav chowdary wrote: for any one who is interested to know about job server from Ooyala.. we started using it recently and been working great so far.. On Feb 25, 2014 9:23 PM, "Ognen Duzlevski" <og...@nengoiksvelzud.com> wrote: > In that case, I must have misunderstood the following (from > http://spark.incubator.apache.org/docs/0.8.1/job-scheduling.html). > Apologies. Ognen > > "Inside a given Spark application (SparkContext instance), multiple > parallel jobs can run simultaneously if they were submitted from separate > threads. By "job", in this section, we mean a Spark action (e.g. save, > collect) and any tasks that need to run to evaluate that action. Spark's > scheduler is fully thread-safe and supports this use case to enable > applications that serve multiple requests (e.g. queries for multiple > users). > > By default, Spark's scheduler runs jobs in FIFO fashion. Each job is > divided into "stages" (e.g. map and reduce phases), and the first job gets > priority on all available resources while its stages have tasks to launch, > then the second job gets priority, etc. If the jobs at the head of the > queue don't need to use the whole cluster, later jobs can start to run > right away, but if the jobs at the head of the queue are large, then later > jobs may be delayed significantly. > > Starting in Spark 0.8, it is also possible to configure fair sharing > between jobs. Under fair sharing, Spark assigns tasks between jobs in a > "round robin" fashion, so that all jobs get a roughly equal share of > cluster resources. This means that short jobs submitted while a long job is > running can start receiving resources right away and still get good > response times, without waiting for the long job to finish. This mode is > best for multi-user settings. > > To enable the fair scheduler, simply set the spark.scheduler.mode to FAIR > before > creating a SparkContext:" > On 2/25/14, 12:30 PM, Mayur Rustagi wrote: > > fair scheduler merely reorders tasks .. I think he is looking to run > multiple pieces of code on a single context on demand from customers...if > the code & order is decided then fair scheduler will ensure that all tasks > get equal cluster time :) > > > > Mayur Rustagi > Ph: +919632149971 > h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com > https://twitter.com/mayur_rustagi > > > > On Tue, Feb 25, 2014 at 10:24 AM, Ognen Duzlevski < > og...@nengoiksvelzud.com> wrote: > >> Doesn't the fair scheduler solve this? >> Ognen >> >> >> On 2/25/14, 12:08 PM, abhinav chowdary wrote: >> >> Sorry for not being clear earlier >> how do you want to pass the operations to the spark context? >> this is partly what i am looking for . How to access the active spark >> context and possible ways to pass operations >> >> Thanks >> >> >> >> On Tue, Feb 25, 2014 at 10:02 AM, Mayur Rustagi <mayur.rust...@gmail.com >> > wrote: >> >>> how do you want to pass the operations to the spark context? >>> >>> >>> Mayur Rustagi >>> Ph: +919632149971 >>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com >>> https://twitter.com/mayur_rustagi >>> >>> >>> >>> On Tue, Feb 25, 2014 at 9:59 AM, abhinav chowdary < >>> abhinav.chowd...@gmail.com> wrote: >>> >>>> Hi, >>>> I am looking for ways to share the sparkContext, meaning i need >>>> to be able to perform multiple operations on the same spark context. >>>> >>>> Below is code of a simple app i am testing >>>> >>>> def main(args: Array[String]) { >>>> println("Welcome to example application!") >>>> >>>> val sc = new SparkContext("spark://10.128.228.142:7077", "Simple >>>> App") >>>> >>>> println("Spark context created!") >>>> >>>> println("Creating RDD!") >>>> >>>> Now once this context is created i want to access this to submit >>>> multiple jobs/operations >>>> >>>> Any help is much appreciated >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>> >> >> >> -- >> Warm Regards >> Abhinav Chowdary >> >> >> > > -- Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. -- Jamie Zawinski
