Hi Gary, What are other frameworks running on your Mesos cluster? If they are all Spark frameworks. Another option you may want to consider (in order to improve your cluster utilization) is to let all of them share a single SparkContext. We also experienced degraded performance while running multiple Spark frameworks on Mesos, mainly due to there are a lot of non-preferred-location tasks.
Best, ~ mbbui On Nov 15, 2013, at 11:30 AM, Gary Malouf <[email protected]> wrote: > Hi everyone, > > I want to share what helped us resolve the issue short term and also our > concerns longer term. > > Some Background: > > Many of our jobs that look at a few weeks of data have task counts around > 3500+. We went with a small cluster of 4 EC2 larges for Mesos+Spark in > production for now because our data sizes are small (3 files hourly on the > order of 10-20mb). The idea is to get this steady and scale horizontally as > the data flow continues to go up. > > Our Problem: > > In short, the time to execute simple tasks with this data of this size for > the past 2 weeks or so was not reasonable for us. The jobs were taking well > over an hour to complete. We were very concerned that as the hourly data > grows in size this will become completely unacceptable. > > The (Short Term) Solution: > > Thanks to Paco and Evan's input, we decided to switch Spark to run in coarse > grained mode for the time-being. We saw an immediate 6-8x improvement on our > job runtime and we were able to complete our short term tasks. > > The downside we see to this is that it takes away many of the benefits of > running on Mesos - namely, the fine-grained resource management. > > The Longer Term Solution: > > We are looking at ways of tuning to reduce the number of Spark tasks being > generated overall. We think if we can get this down enough, it justifies > taking the performance hit in fine-grained mode since the cluster resources > will now be allocated dynamically for multiple jobs. > > > Hope this is helpful to people, > > Gary > > > On Thu, Nov 14, 2013 at 11:47 AM, Gary Malouf <[email protected]> wrote: > I bring this up because the performance we are seeing is dreadful. From cpu > usage, it appears the issue is the spark shell cpu power. We have increased > this node from a EC2 medium to an xl, we are seeing slightly better > performance but still not great. > > My understanding of Spark was that most of the work should be done on the > slaves with just the results going back to the shell at the end if we do a > take. It appears from what we see that the client is doing much more work > than expected. > > > > > On Wed, Nov 13, 2013 at 10:40 PM, Gary Malouf <[email protected]> wrote: > Hi, > > We have an HDFS set up of a namenode and three datanodes all on EC2 larges. > One of our data partitions basically has files that are fed from a few Flume > instances rolling hourly. This equates to around 3 4-8mb files per hour > right now > > Our Mesos cluster consists of a Master and the three slave nodes colocated on > these EC2 larges as well (slaves -> datanodes, mesos master -> namenode). > Spark scheduled jobs are launched from spark shell ad-hoc today. > > The data is serialized protobuf messages in sequence files. Our operations > typically consist of deserializing the data, grabbing a few primitive fields > out of the message and doing some maps/reduces. > > For grabbing on the order of 2 days of data this size, what would the > expected Spark performance be? We are seeing simple maps and 'takes' on this > data taking on the order of 15 minutes. > > Thanks, > > Gary > >
