Spark Mesos Architecture Details and Integration details

Hi All I am a new member of this group. Sorry if this question has already been asked. I am looking for some information regarding Spark Mesos Integration: 1. How does Mesos schedules and launch Spark Executors? (Any pointer to code will be helpful) 2. How does Mesos frontend the Spark

Re: Quota

The dispatcher needs 1cpu and 1G memory. Regards, Vijay Sent from my iPhone > On Dec 9, 2016, at 4:51 PM, Vinod Kone wrote: > > And how many resources does spark need? > >> On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan >> wrote: >>

Re: Quota

And how many resources does spark need? On Fri, Dec 9, 2016 at 4:05 PM, Vijay Srinivasaraghavan < vijikar...@yahoo.com> wrote: > Here is the slave state info. I see marathon is registered as > "slave_public" role and is configured with "default_accepted_resource_roles" > as "*" > > "slaves":[ >

Re: Quota

Here is the slave state info. I see marathon is registered as "slave_public" role and is configured with "default_accepted_resource_roles" as "*" "slaves":[     {    "id":"69356344-e2c4-453d-baaf-22df4a4cc430-S0",  "pid":"slave(1)@xxx.xxx.xxx.100:5051",  

Re: Quota

How many resources does the agent register with the master? How many resources does spark task need? I'm guessing marathon is not registered with "test" role so it is only getting un-reserved resources which are not enough for spark task? On Fri, Dec 9, 2016 at 2:54 PM, Vijay Srinivasaraghavan <

Quota

I have a standalone DCOS setup (Single node Vagrant VM running DCOS v.1.9-dev build + Mesos 1.0.1 + Marathon 1.3.0). Both master and agent are running on same VM. Resource: 4 CPU, 16GB Memory, 20G Disk I have created a quota using new V1 API which creates a role "test" with resource

Re: Duplicate task IDs

Hey Neil, I concur that using duplicate task IDs is bad practice and asking for trouble. Could you please clarify *why* you want to use a hashmap? Is your goal to remove duplicate task IDs or is this just a side-effect and you have a different reason (e.g. performance) for using a hashmap? I'm

Duplicate task IDs

Folks, The master stores a cache of metadata about recently completed tasks; for example, this information can be accessed via the "/tasks" HTTP endpoint or the "GET_TASKS" call in the new Operator API. The master currently stores this metadata using a list; this means that duplicate task IDs

Command healthcheck failed but status KILLED

Hi What is desired behavior when command health check failed? On Mesos 1.0.2 when health check fails task has state KILLED instead of FAILED with reason specifying it was killed due to failing health check. Thanks Tomek

[Review request] Kafka mesos framework pull request #259: Issue 197: --bind-address for broker seems to be ignored

Hi, I created pull request kafka mesos framework #259 to fix issue 197 which was marked as bug by @joe.st...@stealth.ly. Please help to review the change. Any comments please let me know. Thanks, Jared, (??) Software developer Interested in open source