been met.
>
>> On Tue, May 22, 2018 at 2:56 AM, Thodoris Zois wrote:
>> Hello list,
>>
>> I have some questions about resource offers for Mesos and I am
>> experiencing some problems that I hope somebody will be able to help.
>>
>> 1) The allocation mod
o the offers (i.e. not
accepting nor declining them) until your necessary conditions have been met.
On Tue, May 22, 2018 at 2:56 AM, Thodoris Zois wrote:
> Hello list,
>
> I have some questions about resource offers for Mesos and I am
> experiencing some problems that I hope somebod
Hello list,
I have some questions about resource offers for Mesos and I am
experiencing some problems that I hope somebody will be able to help.
1) The allocation module of Mesos master uses DRF (according to
previous allocation history) and decides which framework will get an
offer, and how
t to forward the status immediately and as
a result there can be intermediate offers for resources on the agent before
the reserved resources are re-offered.
Further more it looks like I also sometimes receive a resource offers
> request that only lists my reserved resources but no resourc
Hi,
I'm using Mesos 0.28.2 and my own framework, which uses dynamic
reservations. A task failed and shortly after that I received a resource
offers call that did not contain my reserved resources. I had expected
that the request would already contain my reserved resources that are
no
running on the
>>> same node as the master. Things gets slightly more tricky if your scheduler
>>> is running in a docker container.
>>>
>>> regards,
>>> Hendrik
>>>
>>>> On 27.09.2016 14:34, Eli Jordan wrote:
>>>> Y
ordan wrote:
Yes, it appears in the mesos ui, and stays there. I log all messages from the
mesos master, including resource offers and disconnected. I don't receive
offers or disconnected.
I know I need to accept or decline the offers, the problem is that I never
receive the resource offe
ainer.
>
> regards,
> Hendrik
>
>> On 27.09.2016 14:34, Eli Jordan wrote:
>> Yes, it appears in the mesos ui, and stays there. I log all messages from
>> the mesos master, including resource offers and disconnected. I don't
>> receive offers or disconnecte
on the same node as the master. Things gets slightly more tricky if your
scheduler is running in a docker container.
regards,
Hendrik
On 27.09.2016 14:34, Eli Jordan wrote:
Yes, it appears in the mesos ui, and stays there. I log all messages
from the mesos master, including resource offers and
Yes, it appears in the mesos ui, and stays there. I log all messages from the
mesos master, including resource offers and disconnected. I don't receive
offers or disconnected.
I know I need to accept or decline the offers, the problem is that I never
receive the resource offer, but the m
On 09/27/2016 02:08 PM, Gmail wrote:
> Hi
>
> I am implementing a mesos framework, and have hit a strange issue that I
> can't make sense of. Intermittently, my framework will receive the registered
> message, and is shown as registered in the mesos ui.
>
> I never see any resource offer messa
Hi
I am implementing a mesos framework, and have hit a strange issue that I can't
make sense of. Intermittently, my framework will receive the registered
message, and is shown as registered in the mesos ui.
I never see any resource offer messages being processed by the framework,
however, the
<mailto:hendrik.hadd...@gmx.net>
> <mailto:hendrik.hadd...@gmx.net
> <mailto:hendrik.hadd...@gmx.net>>>> wrote:
>
> Hi,
>
> I have three Mesos cl
cline offer? If you are enabling GLOG_v=2 for
>>> mesos master, you will get some log as "Framework xxx filtered
>>> agent for "
>>>
>>> On Mon, Sep 26, 2016 at 2:47 PM, Hendrik Haddorp
>>> mailto:hendrik.hadd...@gmx.net
hadd...@gmx.net>
<mailto:hendrik.hadd...@gmx.net
<mailto:hendrik.hadd...@gmx.net>>> wrote:
Hi,
I have three Mesos cluster test setups. On two my
frameworks gets
the resource offers from all slaves in one
"resourceOffe
,
I have three Mesos cluster test setups. On two my
frameworks gets
the resource offers from all slaves in one
"resourceOffers" call.
In one three node setup I do however sometimes get offers
for all
slaves but most of t
ered agent for "
>>
>> On Mon, Sep 26, 2016 at 2:47 PM, Hendrik Haddorp > <mailto:hendrik.hadd...@gmx.net>> wrote:
>>
>> Hi,
>>
>> I have three Mesos cluster test setups. On two my frameworks gets
>> the resource offers from all
rks gets
the resource offers from all slaves in one "resourceOffers" call.
In one three node setup I do however sometimes get offers for all
slaves but most of the time I get first two offers and then the
third in a separate call. Should I get all offers in one call or
do I
k xxx
filtered agent for "
On Mon, Sep 26, 2016 at 2:47 PM, Hendrik Haddorp
wrote:
> Hi,
>
> I have three Mesos cluster test setups. On two my frameworks gets the
> resource offers from all slaves in one "resourceOffers" call. In one three
> node setup I do however some
Hi,
I have three Mesos cluster test setups. On two my frameworks gets the
resource offers from all slaves in one "resourceOffers" call. In one
three node setup I do however sometimes get offers for all slaves but
most of the time I get first two offers and then the third in a sepa
Ah, thanks for the clarification. I can't find any logs from the framework
indicating that we got the initial offer, so it looks like it could have
been dropped. We haven't set --offer-timeout on our masters, so your
explanation makes sense. Thanks!
On Mon, Apr 25, 2016 at 4:17 PM, Vinod Kone wro
> I0421 21:03:32.014999 17071 master.cpp:4290] Sending 1 offers to
> framework sy3x4 (sy3x4) at
> scheduler-6bb2bcf0-d060-4072-a25b-917d8007fb1c@172.16.13.243:56861
>
This shows that the slaves resources were sent to a framework. Looks like
the framework is holding on to the offer for a long time?
I0421 21:03:32.014533 17073 hierarchical.hpp:528] Added slave
20151116-203437-35000492-5050-17068-S70 (lively-rice) with mem(*):217609;
cpus(*):210; ports(*):[2048-3048]; disk(*):639829 (allocated: )
I0421 21:03:32.014529 17072 master.cpp:3395] Registered slave
20151116-203437-35000492-5050-17068-S
On Mon, Apr 25, 2016 at 8:40 AM, Thomas Petr wrote:
> The only thing that ended up fixing the situation was bouncing our
> scheduler (~10 minutes after the restarted slaves joined the cluster) --
> the act of failing over the framework appeared to "recover" the missing
> resources:
>
What do the
03437-35000492-5050-17068-S70
I0421 21:03:32.016317 53215 status_update_manager.cpp:183] Resuming
sending status updates
Everything makes perfect sense up to this point. The slaves appear to be
online and connected to the cluster, but we quickly noticed that these
slaves were not sending resource of
eview :).
>>>
>>>
>>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>>
>>>
:).
>>
>>
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>>
>> --
>> Date: Tue, 29 Mar 201
gt;
>
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
>
> --
> Date: Tue, 29 Mar 2016 10:51:44 +0100
> Subject: P
+86-10-8245 4084 | klaus1982...@gmail.com |
http://k82.me
Date: Tue, 29 Mar 2016 10:51:44 +0100
Subject: Port Resource Offers
From: pradeep.chhetr...@gmail.com
To: user@mesos.apache.org
Hello,
I am running mesos slaves with the modified port announcement.
$ cat /etc/mesos-slave/resourcesports
Hello,
I am running mesos slaves with the modified port announcement.
$ cat /etc/mesos-slave/resources
ports(*):[6379, 9200, 9300, 27017, 31000-35000]
I can that this is being picked up when starting the mesos slaves in ps
output:
--resources=ports(*):[6379, 9200, 9300, 27017, 31000-35000]
How
Wanted to add that, even if there wasn’t a preview package, you can clone from
GIT, and checkout a tag, where in this case v1.5.0-rc1 is tagged. Then
proceeded normally as you would’ve had a source distro as described in the
already mentioned http://spark.apache.org/docs/latest/building-spark.ht
THANKS, as I have not kept up on the spark lists
James
On 08/25/2015 04:28 AM, Iulian Dragoș wrote:
On Mon, Aug 24, 2015 at 7:16 PM, CCAAT mailto:cc...@tampabay.rr.com>> wrote:
On 08/24/2015 05:33 AM, Iulian Dragoș wrote:
Hello Iulian,
Ok, so I eventually build spark from
On Mon, Aug 24, 2015 at 7:16 PM, CCAAT wrote:
> On 08/24/2015 05:33 AM, Iulian Dragoș wrote:
>
>
> Hello Iulian,
>
> Ok, so I eventually build spark from 100% sources, after some intermediate
> builds on gentoo. Gentoo is not the best platform for Java development,
> but those issues related to
On 08/24/2015 05:33 AM, Iulian Dragoș wrote:
Hello Iulian,
Ok, so I eventually build spark from 100% sources, after some
intermediate builds on gentoo. Gentoo is not the best platform for
Java development, but those issues related to spark builds are slowly
being fixed on gentoo. Where (ho
chance of getting the same amount of
> memory, but as their dominant resource is lower (memory) they more often
> get CPU resources compared to that first instance. Counter intuitively, the
> first instance finishes last.
>
> On 19 Aug 2015, at 14:07, Iulian Dragoș
> wrote:
>
&
t getting any resource offers for
> 15-20 minutes, while other frameworks (8-9 of them) continuously get offers.
>
> The framework is Spark (running in fine-grained mode), and is launched with
> Chronos. After a few tasks successfully executed, it stops getting offers,
> though lo
I am facing a problem with a framework not getting any resource offers for
15-20 minutes, while other frameworks (8-9 of them) continuously get offers.
The framework is Spark (running in fine-grained mode), and is launched with
Chronos. After a few tasks successfully executed, it stops getting
rs, it won't be re-offered the
>>>> same resources for some period of time.
>>>>
>>>> On Sat, Jun 13, 2015 at 8:30 PM, Ondrej Smola
>>>> wrote:
>>>> Hi Christopher,
>>>>
>>>> i dont know about any way way how
mola
>>>> wrote:
>>>>
>>>>> Hi Christopher,
>>>>>
>>>>> i dont know about any way way how to speed up first resource offer -
>>>>> in my experience new offers arrive almost immediately after framework
>>>&
registration. It depends on the infrastructure you are testing your
>>> framework on - are there any
>>> other frameworks running? As is discussed in an another thread offers
>>> should be send to multiple frameworks at once. There may be small
>>> delay based on i
ng your
>>>> framework on - are there any
>>>> other frameworks running? As is discussed in an another thread offers
>>>> should be send to multiple frameworks at once. There may be small
>>>> delay based on initial registration and network delay. If
multiple frameworks at once. There may be small
>> delay based on initial registration and network delay. If you speak
>> about "reoffers" - reoffering
>> decline offers - there should param to set interval for reoffer. For
>> example in Go you can decline offer
rs
>>>> should be send to multiple frameworks at once. There may be small
>>>> delay based on initial registration and network delay. If you speak
>>>> about "reoffers" - reoffering
>>>> decline offers - there should param to set interval for
ld param to set interval for reoffer. For
>>> example in Go you can decline offer this way (it is also important to
>>> decline every non used offer):
>>>
>>> driver.DeclineOffer(offer.Id, &mesos.Filters{RefuseSeconds:
>>> proto.Float64(5)})
>>&g
p;mesos.Filters{RefuseSeconds: proto.Float64(5)})
>
> Look to mesos UI - it shoud give you information abou what offers are
> offered to which frameworks, mesos master logs also give you this
> information.
>
>
> 2015-06-13 18:23 GMT+02:00 Christopher Ketchum <mailto:cketc...
UI - it shoud give you information abou what offers are
>> offered to which frameworks, mesos master logs also give you this
>> information.
>>
>>
>> 2015-06-13 18:23 GMT+02:00 Christopher Ketchum :
>> > Hi,
>> >
>> > I was wondering if there
d give you information abou what offers are
> offered to which frameworks, mesos master logs also give you this
> information.
>
>
> 2015-06-13 18:23 GMT+02:00 Christopher Ketchum :
> > Hi,
> >
> > I was wondering if there was any way to adjust the rate of resource
&g
ion.
2015-06-13 18:23 GMT+02:00 Christopher Ketchum :
> Hi,
>
> I was wondering if there was any way to adjust the rate of resource offers to
> the framework. I am writing a mesos framework, and when I am testing it I am
> noticing a slight pause were the framework seems to be wa
Hi,
I was wondering if there was any way to adjust the rate of resource offers to
the framework. I am writing a mesos framework, and when I am testing it I am
noticing a slight pause were the framework seems to be waiting for another
resource offer. I would like to know if there is any way to
flags when you launch a mesos slave. And when this slave's resources is
>>>> being offered, it will also include all the attributes you've tagged.
>>>>
>>>> This currently is static information on launch, and I believe there is
>>>> JIRA tic
static information on launch, and I believe there is
>>> JIRA tickets to make this dynamic (updatable at runtime).
>>>
>>> Tim
>>>
>>> On Thu, Jan 15, 2015 at 7:23 PM, Douglas Voet
>>> wrote:
>>>
>>>> Hello,
>>>>
lude all the attributes you've tagged.
>>>
>>> This currently is static information on launch, and I believe there is
>>> JIRA tickets to make this dynamic (updatable at runtime).
>>>
>>> Tim
>>>
>>> On Thu, Jan 15, 2015 at 7:23 PM, D
t; Tim
>>
>> On Thu, Jan 15, 2015 at 7:23 PM, Douglas Voet
>> wrote:
>>
>>> Hello,
>>>
>>> I am evaluating mesos in the context of running analyses of many large
>>> files. I only want to download a file to a small subset of my nodes and
>&
file to a small subset of my nodes and
>> route the related processing there. The mesos paper talks about using
>> resource offers as a mechanism to achieve data locality but I can't find
>> any reference to how one might do this in the documentation. How would a
>> meso
nd
> route the related processing there. The mesos paper talks about using
> resource offers as a mechanism to achieve data locality but I can't find
> any reference to how one might do this in the documentation. How would a
> mesos slave know what data is available keeping in mind that t
Hello,
I am evaluating mesos in the context of running analyses of many large
files. I only want to download a file to a small subset of my nodes and
route the related processing there. The mesos paper talks about using
resource offers as a mechanism to achieve data locality but I can't fin
ded
>> framework 20141106-193147-16842879-5050-10406-
>> I1106 19:32:29.647886 10423 http.cpp:391] HTTP request for
>> '/master/state.json'
>>
>>
>> On Thu, Nov 6, 2014 at 6:53 PM, Benjamin Mahler <
>> benjamin.mah...@gmail.com>
h version of the master are you using and do you have the logs? The
>> fact that no offers were coming back sounds like a bug!
>>
>> As for using O1 after a disconnection, all offers are invalid once a
>> disconnection occurs. The scheduler driver does not automatically rescind
ing back sounds like a bug!
>>
>> As for using O1 after a disconnection, all offers are invalid once a
>> disconnection occurs. The scheduler driver does not automatically rescind
>> offers upon disconnection, so I'd recommend clearing all cached offers when
>&
ST
> updates.
>
> On Thu, Nov 6, 2014 at 6:25 PM, Sharma Podila wrote:
>
>> We had an interesting problem with resource offers today and I would like
>> to confirm this problem and request an enhancement. Here's the summary in
>> the right sequence of events:
>&g
, so I'd recommend clearing all cached offers when
your scheduler gets disconnected, to avoid the unnecessary TASK_LOST
updates.
On Thu, Nov 6, 2014 at 6:25 PM, Sharma Podila wrote:
> We had an interesting problem with resource offers today and I would like
> to confirm this problem and
We had an interesting problem with resource offers today and I would like
to confirm this problem and request an enhancement. Here's the summary in
the right sequence of events:
1. resource offer O1 for slave A arrives
2. mesos disconnects
3. mesos reregisters
4. mesos offer O2 for sl
>
> I'm not sure these two cases are any different. The TASK_INVALID_OFFER
> would model a terminal state for the task. Afterwards, one still has to
> generate a new "TaskInfo" in so far as the TaskID should not be re-used
> across launch requests.
I was expecting to reuse the TaskID. If it can't
>
>
> (1) If the slave is unknown, we send TASK_LOST.
> (2) If the task is missing on the slave, we send TASK_LOST.
> (3) If the task state differs, we send the latest state.
> In the absence of bugs or data loss, (1) is the only one that is strictly
> necessary for correctness. In your case,
Thanks for providing more details!
I'm not sure these two cases are any different. The TASK_INVALID_OFFER
would model a terminal state for the task. Afterwards, one still has to
generate a new "TaskInfo" in so far as the TaskID should not be re-used
across launch requests.
*For example, what if r
>
> Where as, a TASK_LOST will make me (unnecessarily, in this case) try to
> ensure that the task is actually lost, not running away on the slave that
> got disconnected from Mesos master. Not all environments may need the
> distinction, but at least some do.
To be clear, are you still planning
TASK_LOST is a good thing. I expect to deal with it now and in the future.
I was trying to distinguish this:
- case TASK_LOST:
- persist state update to TASK_LOST
- create new task submission request
- schedule with next available offer
- case TASK_INVALID_OFFER:
- pe
Thanks for confirming that, Adam.
> , but it would be a good Mesos FAQ topic.
I was thinking it might be good to also add to doc in code, either in
mesos.proto or MesosSchedulerDriver (mesos.proto already refers to the
latter for failover at FrameworkID definition).
If you were to try to pers
Correct, Sharma. I don't think this is documented anywhere yet, but it
would be a good Mesos FAQ topic.
When the master notices that the framework has exited or is deactivated, it
disables the framework in the allocator so no new offers will be made to
that framework, and removes any outstanding of
My understanding is that when a framework fails over (either new instance
starts after previous one fails, or the same instance restarts), Mesos
master would automatically cancel any unused offers it had given to the
previous framework instance. This is a good thing. Can someone confirm this
to be
70 matches
Mail list logo
