Hi, I have several instances of the same web-service that is running some ML algos on Spark (both training and prediction) and do some Spark unrelated job. Each web-service instance creates their on JavaSparkContext, thus they're seen as separate applications by Spark, thus they're configured with separate limits of resources such as cores (I'm not concerned about the memory as much as about cores).
With this set up, say 3 web service instances, each of them has just 1/3 of cores. But it might happen, than only one instance is going to use Spark, while others are busy with Spark unrelated. I'd like in this case all Spark cores be available for the one that's in need. Ideally I'd like Spark cores just be available in total and the first app who needs it, takes as much as required from the available at the moment. Is it possible? I believe Mesos is able to set resources free if they're not in use. Is it possible with YARN? I'd appreciate if you could share your thoughts or experience on the subject. Thanks. -- Be well! Jean Morozov
