[jira] [Commented] (MESOS-4436) Propose design doc for fixed-point scalar resources
[ https://issues.apache.org/jira/browse/MESOS-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392568#comment-15392568 ] Erik Weathers commented on MESOS-4436: -- [~neilc]: aha! I had looked at the fixed-point-conversion code in the referenced ticket and so that's why I thought this was a DUP. Thanks for the info on what this was intended for, I'll change it back to "won't fix". > Propose design doc for fixed-point scalar resources > --- > > Key: MESOS-4436 > URL: https://issues.apache.org/jira/browse/MESOS-4436 > Project: Mesos > Issue Type: Task > Components: general >Reporter: Neil Conway > Labels: mesosphere, resources > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3341) Introduce Resource Resolution
[ https://issues.apache.org/jira/browse/MESOS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390317#comment-15390317 ] Erik Weathers commented on MESOS-3341: -- Just spoke with [~jessicalhartog] about this, we believe that MESOS-4687 has fixed this issue. So I'm resolving this one. > Introduce Resource Resolution > - > > Key: MESOS-3341 > URL: https://issues.apache.org/jira/browse/MESOS-3341 > Project: Mesos > Issue Type: Improvement >Reporter: Jessica Hartog >Priority: Minor > > After MESOS-1807, Mesos containers require >= 0.01 CPU resources and >= 32MB > Memory resources. In order to simplify accounting, Mesos should introduce > resource resolution. > For example, it is possible to launch a task with 1.1 CPU (as it > exceeds the minimum number of CPUs and is therefore considered valid). The > fractional component of this task does not benefit the running process, and > can introduce floating point errors when Mesos is accounting its offers > (which we have already seen happening in MESOS-1867 and MESOS-2635). A > solution to this could be disallowing tasks with finer granularity than the > required resolution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4436) Propose design doc for fixed-point scalar resources
[ https://issues.apache.org/jira/browse/MESOS-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390316#comment-15390316 ] Erik Weathers commented on MESOS-4436: -- Isn't this a duplicate of MESOS-4545? Rather than "won't fix"? > Propose design doc for fixed-point scalar resources > --- > > Key: MESOS-4436 > URL: https://issues.apache.org/jira/browse/MESOS-4436 > Project: Mesos > Issue Type: Task > Components: general >Reporter: Neil Conway > Labels: mesosphere, resources > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4735) CommandInfo.URI should allow specifying target filename
[
https://issues.apache.org/jira/browse/MESOS-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245149#comment-15245149
]
Erik Weathers commented on MESOS-4735:
--
[~mrbrowning] & [~vinodkone]: awesome, thanks so much for fixing this!
> CommandInfo.URI should allow specifying target filename
> ---
>
> Key: MESOS-4735
> URL: https://issues.apache.org/jira/browse/MESOS-4735
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Reporter: Erik Weathers
>Assignee: Michael Browning
>Priority: Minor
> Fix For: 0.29.0
>
>
> The {{CommandInfo.URI}} message should allow explicitly choosing the
> downloaded file's name, to better mimic functionality present in tools like
> {{wget}} and {{curl}}.
> This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
> has query parameters at the end of the path, resulting in the downloaded
> filename having those elements. This also prevents extracting of such files,
> since the extraction logic is simply looking at the file's suffix. See
> MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was
> fixed, then I could workaround the other issues not being fixed by modifying
> my framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3367) Mesos fetcher does not extract archives for URI with parameters
[ https://issues.apache.org/jira/browse/MESOS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245147#comment-15245147 ] Erik Weathers commented on MESOS-3367: -- [~haosd...@gmail.com]: yup, that looks good to me! It should suffice from my perspective. Not sure about [~bernd-mesos] & [~zubairov] though. > Mesos fetcher does not extract archives for URI with parameters > --- > > Key: MESOS-3367 > URL: https://issues.apache.org/jira/browse/MESOS-3367 > Project: Mesos > Issue Type: Improvement > Components: fetcher >Affects Versions: 0.22.1, 0.23.0 > Environment: DCOS 1.1 >Reporter: Renat Zubairov >Assignee: haosdent >Priority: Minor > Labels: mesosphere > > I'm deploying using marathon applications with sources served from S3. I'm > using a signed URL to give only temporary access to the S3 resources, so URL > of the resource have some query parameters. > So URI is 'https://foo.com/file.tgz?hasi' and fetcher stores it in the file > with the name 'file.tgz?hasi', then it thinks that extension 'hasi' is not > tgz hence extraction is skipped, despite the fact that MIME Type of the HTTP > resource is 'application/x-tar'. > Workaround - add additional parameter like '=.tgz' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4735) CommandInfo.URI should allow specifying target filename
[
https://issues.apache.org/jira/browse/MESOS-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174464#comment-15174464
]
Erik Weathers commented on MESOS-4735:
--
[~dma1982] : kind of correct.. I'm saying that people that use file
downloading tools (e.g., curl, wget, every single web browser) have the option
to choose the downloaded resultant filename. e.g.,
* {{curl -o bar-executor-binary.tgz
http://somewebserver/bar-executor-binary.tgz.foobarbazblahblahblah}}
* {{wget -O bar-executor-binary.tgz
http://somewebserver/bar-executor-binary.tgz.foobarbazblahblahblah}}
> CommandInfo.URI should allow specifying target filename
> ---
>
> Key: MESOS-4735
> URL: https://issues.apache.org/jira/browse/MESOS-4735
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Reporter: Erik Weathers
>Assignee: Guangya Liu
>Priority: Minor
>
> The {{CommandInfo.URI}} message should allow explicitly choosing the
> downloaded file's name, to better mimic functionality present in tools like
> {{wget}} and {{curl}}.
> This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
> has query parameters at the end of the path, resulting in the downloaded
> filename having those elements. This also prevents extracting of such files,
> since the extraction logic is simply looking at the file's suffix. See
> MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was
> fixed, then I could workaround the other issues not being fixed by modifying
> my framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (MESOS-4735) CommandInfo.URI should allow specifying target filename
[
https://issues.apache.org/jira/browse/MESOS-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173519#comment-15173519
]
Erik Weathers edited comment on MESOS-4735 at 3/1/16 9:57 AM:
--
[~gyliu], MESOS-3367 fixes one of the issues I appended above, but not the
other. This proposal is more general than either of those issues, providing
some level of future-proofness against other unforeseen issues.
Ignoring those other issues, the Mesos fetcher is acting as an HTTP downloader,
and all of the utilities I use on a day-to-day basis for that already support
the functionality this ticket is requesting: choosing the filename to save the
downloaded file as. Browsers let you do that, as do {{curl}} and {{wget}}. So
it's just something that should be added sooner or later to the Mesos fetcher,
and the fact that this would allow for other various problems to be overcome by
a framework author is just another benefit.
was (Author: erikdw):
[~gyliu] MESOS-3367 fixes one of the issues I appended above, but not the
other. This proposal is more general than either of those issues, providing
some level of future-proofness against other unforeseen issues.
Ignoring those other issues, the Mesos fetcher is acting as an HTTP downloader,
and all of the utilities I use on a day-to-day basis for that already support
the functionality this ticket is requesting: choosing the filename to save the
downloaded file as. Browsers let you do that, as do {{curl}} and {{wget}}. So
it's just something that should be added sooner or later to the Mesos fetcher,
and the fact that this would allow for other various problems to be overcome by
a framework author is just another benefit.
> CommandInfo.URI should allow specifying target filename
> ---
>
> Key: MESOS-4735
> URL: https://issues.apache.org/jira/browse/MESOS-4735
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Affects Versions: 0.27.0
>Reporter: Erik Weathers
>Assignee: Guangya Liu
>Priority: Minor
>
> The {{CommandInfo.URI}} message should allow explicitly choosing the
> downloaded file's name, to better mimic functionality present in tools like
> {{wget}} and {{curl}}.
> This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
> has query parameters at the end of the path, resulting in the downloaded
> filename having those elements. This also prevents extracting of such files,
> since the extraction logic is simply looking at the file's suffix. See
> MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was
> fixed, then I could workaround the other issues not being fixed by modifying
> my framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4735) CommandInfo.URI should allow specifying target filename
[
https://issues.apache.org/jira/browse/MESOS-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173519#comment-15173519
]
Erik Weathers commented on MESOS-4735:
--
[~gyliu] MESOS-3367 fixes one of the issues I appended above, but not the
other. This proposal is more general than either of those issues, providing
some level of future-proofness against other unforeseen issues.
Ignoring those other issues, the Mesos fetcher is acting as an HTTP downloader,
and all of the utilities I use on a day-to-day basis for that already support
the functionality this ticket is requesting: choosing the filename to save the
downloaded file as. Browsers let you do that, as do {{curl}} and {{wget}}. So
it's just something that should be added sooner or later to the Mesos fetcher,
and the fact that this would allow for other various problems to be overcome by
a framework author is just another benefit.
> CommandInfo.URI should allow specifying target filename
> ---
>
> Key: MESOS-4735
> URL: https://issues.apache.org/jira/browse/MESOS-4735
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Affects Versions: 0.27.0
>Reporter: Erik Weathers
>Assignee: Guangya Liu
>Priority: Minor
>
> The {{CommandInfo.URI}} message should allow explicitly choosing the
> downloaded file's name, to better mimic functionality present in tools like
> {{wget}} and {{curl}}.
> This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
> has query parameters at the end of the path, resulting in the downloaded
> filename having those elements. This also prevents extracting of such files,
> since the extraction logic is simply looking at the file's suffix. See
> MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was
> fixed, then I could workaround the other issues not being fixed by modifying
> my framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3367) Mesos fetcher does not extract archives for URI with parameters
[ https://issues.apache.org/jira/browse/MESOS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173132#comment-15173132 ] Erik Weathers commented on MESOS-3367: -- [~xujyan]: thanks for adding the response. I filed MESOS-4735 for specifying the output/result filename. > Mesos fetcher does not extract archives for URI with parameters > --- > > Key: MESOS-3367 > URL: https://issues.apache.org/jira/browse/MESOS-3367 > Project: Mesos > Issue Type: Improvement > Components: fetcher >Affects Versions: 0.22.1, 0.23.0 > Environment: DCOS 1.1 >Reporter: Renat Zubairov >Assignee: haosdent >Priority: Minor > Labels: mesosphere > > I'm deploying using marathon applications with sources served from S3. I'm > using a signed URL to give only temporary access to the S3 resources, so URL > of the resource have some query parameters. > So URI is 'https://foo.com/file.tgz?hasi' and fetcher stores it in the file > with the name 'file.tgz?hasi', then it thinks that extension 'hasi' is not > tgz hence extraction is skipped, despite the fact that MIME Type of the HTTP > resource is 'application/x-tar'. > Workaround - add additional parameter like '=.tgz' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4735) CommandInfo.URI should allow specifying target filename
[
https://issues.apache.org/jira/browse/MESOS-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173129#comment-15173129
]
Erik Weathers commented on MESOS-4735:
--
[~gyliu]: sure, I'm talking about the filename within the executor sandbox.
e.g., say that the {{CommandInfo.URI.value}} has the following URL:
{code: title=Example CommandInfo.URI.value}
http://hadoop-namenode.com:50070/webhdfs/v1/user/foo/bar-executor-binary.tgz?op=OPEN
{code}
In that case, using the current mesos fetcher behavior as of mesos-0.27.0, the
downloaded file in the executor sandbox would be:
{code: title=Example Current Result Filename}
bar-executor-binary.tgz?op=OPEN
{code}
However, I would like to be able to override this result filename in the
executor sandbox to be something like:
{code: title=Desired Result Filename}
bar-executor-binary.tgz
{code}
Benefits:
* allows explicit avoidance of inclusion of query parameters from the URI in
the filename (see MESOS-1686)
* the fetcher's extracting logic could actually extract such executor bundles
(tar, zip, etc.) (see MESOS-3367)
> CommandInfo.URI should allow specifying target filename
> ---
>
> Key: MESOS-4735
> URL: https://issues.apache.org/jira/browse/MESOS-4735
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Affects Versions: 0.27.0
>Reporter: Erik Weathers
>Assignee: Guangya Liu
>Priority: Minor
>
> The {{CommandInfo.URI}} message should allow explicitly choosing the
> downloaded file's name, to better mimic functionality present in tools like
> {{wget}} and {{curl}}.
> This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
> has query parameters at the end of the path, resulting in the downloaded
> filename having those elements. This also prevents extracting of such files,
> since the extraction logic is simply looking at the file's suffix. See
> MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was
> fixed, then I could workaround the other issues not being fixed by modifying
> my framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4675) Cannot disable systemd support
[ https://issues.apache.org/jira/browse/MESOS-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Weathers updated MESOS-4675: - Summary: Cannot disable systemd support (was: Can not disable systemd support) > Cannot disable systemd support > -- > > Key: MESOS-4675 > URL: https://issues.apache.org/jira/browse/MESOS-4675 > Project: Mesos > Issue Type: Bug >Affects Versions: 0.25.0, 0.26.0, 0.27.0 >Reporter: Joris Van Remoortere >Assignee: Joris Van Remoortere > Labels: mesosphere, systemd > Fix For: 0.27.1, 0.28.0 > > > On certain platforms the systemd init system is available, but not used. > Not being able to disable the mesos systemd integration on these platforms > makes it hard to operate using a different init / monit system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4737) document TaskID uniqueness requirement
[
https://issues.apache.org/jira/browse/MESOS-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Weathers updated MESOS-4737:
-
Attachment: Reusing Task IDs.pdf
> document TaskID uniqueness requirement
> --
>
> Key: MESOS-4737
> URL: https://issues.apache.org/jira/browse/MESOS-4737
> Project: Mesos
> Issue Type: Task
> Components: documentation
>Affects Versions: 0.27.0
>Reporter: Erik Weathers
>Assignee: Erik Weathers
>Priority: Minor
> Labels: documentation
> Attachments: Reusing Task IDs.pdf
>
>
> There are comments above the definition of TaskID in
> [mesos.proto|https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66]
> which lead one to believe it is ok to reuse TaskID values so long as you
> guarantee there will only ever be 1 such TaskID running at the same time.
> {code: title=existing comments for TaskID}
> * A framework generated ID to distinguish a task. The ID must remain
> * unique while the task is active. However, a framework can reuse an
> * ID _only_ if a previous task with the same ID has reached a
> * terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).
> {code}
> However, there are a few scenarios where problems can arise.
> # The checkpointing-and-recovery feature of mesos-slave/agent clashes with
> tasks that reuse an ID and get assigned to the same executor.
> #* See [this
> email|https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAO5KYW8%2BXMWc1dXtEo20BAsfGow028jwjL2ubMinP%2BK%2BvdOh8w%40mail.gmail.com%3E]
> for more info, as well as the attachment on this issue.
> # Issues during network partitions and master failover, where a TaskID might
> appear to be unique in the system, whereas in actuality another Task is
> running with that ID and was just partitioned away for some time.
> In light of these issues, we should simply update the document(s) to make it
> abundantly clear that reusing TaskIDs is never ok. At the minimum this
> should involve updating the afore-mentioned comments in {{mesos.proto}}.
> Also any framework development guides that talk about TaskID creation should
> be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-4737) document TaskID uniqueness requirement
Erik Weathers created MESOS-4737:
Summary: document TaskID uniqueness requirement
Key: MESOS-4737
URL: https://issues.apache.org/jira/browse/MESOS-4737
Project: Mesos
Issue Type: Task
Components: documentation
Affects Versions: 0.27.0
Reporter: Erik Weathers
Assignee: Erik Weathers
Priority: Minor
There are comments above the definition of TaskID in
[mesos.proto|https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66]
which lead one to believe it is ok to reuse TaskID values so long as you
guarantee there will only ever be 1 such TaskID running at the same time.
{code title=existing comments for TaskID}
* A framework generated ID to distinguish a task. The ID must remain
* unique while the task is active. However, a framework can reuse an
* ID _only_ if a previous task with the same ID has reached a
* terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).
{code}
However, there are a few scenarios where problems can arise.
# The checkpointing-and-recovery feature of mesos-slave/agent clashes with
tasks that reuse an ID and get assigned to the same executor.
#* See [this
email|https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAO5KYW8%2BXMWc1dXtEo20BAsfGow028jwjL2ubMinP%2BK%2BvdOh8w%40mail.gmail.com%3E]
for more info, as well as the attachment on this issue.
# Issues during network partitions and master failover, where a TaskID might
appear to be unique in the system, whereas in actuality another Task is running
with that ID and was just partitioned away for some time.
In light of these issues, we should simply update the document(s) to make it
abundantly clear that reusing TaskIDs is never ok. At the minimum this should
involve updating the afore-mentioned comments in {{mesos.proto}}. Also any
framework development guides that talk about TaskID creation should be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4737) document TaskID uniqueness requirement
[
https://issues.apache.org/jira/browse/MESOS-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Weathers updated MESOS-4737:
-
Description:
There are comments above the definition of TaskID in
[mesos.proto|https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66]
which lead one to believe it is ok to reuse TaskID values so long as you
guarantee there will only ever be 1 such TaskID running at the same time.
{code: title=existing comments for TaskID}
* A framework generated ID to distinguish a task. The ID must remain
* unique while the task is active. However, a framework can reuse an
* ID _only_ if a previous task with the same ID has reached a
* terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).
{code}
However, there are a few scenarios where problems can arise.
# The checkpointing-and-recovery feature of mesos-slave/agent clashes with
tasks that reuse an ID and get assigned to the same executor.
#* See [this
email|https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAO5KYW8%2BXMWc1dXtEo20BAsfGow028jwjL2ubMinP%2BK%2BvdOh8w%40mail.gmail.com%3E]
for more info, as well as the attachment on this issue.
# Issues during network partitions and master failover, where a TaskID might
appear to be unique in the system, whereas in actuality another Task is running
with that ID and was just partitioned away for some time.
In light of these issues, we should simply update the document(s) to make it
abundantly clear that reusing TaskIDs is never ok. At the minimum this should
involve updating the afore-mentioned comments in {{mesos.proto}}. Also any
framework development guides that talk about TaskID creation should be updated.
was:
There are comments above the definition of TaskID in
[mesos.proto|https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66]
which lead one to believe it is ok to reuse TaskID values so long as you
guarantee there will only ever be 1 such TaskID running at the same time.
{code title=existing comments for TaskID}
* A framework generated ID to distinguish a task. The ID must remain
* unique while the task is active. However, a framework can reuse an
* ID _only_ if a previous task with the same ID has reached a
* terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).
{code}
However, there are a few scenarios where problems can arise.
# The checkpointing-and-recovery feature of mesos-slave/agent clashes with
tasks that reuse an ID and get assigned to the same executor.
#* See [this
email|https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAO5KYW8%2BXMWc1dXtEo20BAsfGow028jwjL2ubMinP%2BK%2BvdOh8w%40mail.gmail.com%3E]
for more info, as well as the attachment on this issue.
# Issues during network partitions and master failover, where a TaskID might
appear to be unique in the system, whereas in actuality another Task is running
with that ID and was just partitioned away for some time.
In light of these issues, we should simply update the document(s) to make it
abundantly clear that reusing TaskIDs is never ok. At the minimum this should
involve updating the afore-mentioned comments in {{mesos.proto}}. Also any
framework development guides that talk about TaskID creation should be updated.
> document TaskID uniqueness requirement
> --
>
> Key: MESOS-4737
> URL: https://issues.apache.org/jira/browse/MESOS-4737
> Project: Mesos
> Issue Type: Task
> Components: documentation
>Affects Versions: 0.27.0
>Reporter: Erik Weathers
>Assignee: Erik Weathers
>Priority: Minor
> Labels: documentation
>
> There are comments above the definition of TaskID in
> [mesos.proto|https://github.com/apache/mesos/blob/0.27.0/include/mesos/mesos.proto#L63-L66]
> which lead one to believe it is ok to reuse TaskID values so long as you
> guarantee there will only ever be 1 such TaskID running at the same time.
> {code: title=existing comments for TaskID}
> * A framework generated ID to distinguish a task. The ID must remain
> * unique while the task is active. However, a framework can reuse an
> * ID _only_ if a previous task with the same ID has reached a
> * terminal state (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.).
> {code}
> However, there are a few scenarios where problems can arise.
> # The checkpointing-and-recovery feature of mesos-slave/agent clashes with
> tasks that reuse an ID and get assigned to the same executor.
> #* See [this
> email|https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3CCAO5KYW8%2BXMWc1dXtEo20BAsfGow028jwjL2ubMinP%2BK%2BvdOh8w%40mail.gmail.com%3E]
> for more info, as well as the attachment on this issue.
> # Issues during network partitions and master failover, where a TaskID might
> appear
[jira] [Created] (MESOS-4735) CommandInfo.URI should allow specifying target filename
Erik Weathers created MESOS-4735:
Summary: CommandInfo.URI should allow specifying target filename
Key: MESOS-4735
URL: https://issues.apache.org/jira/browse/MESOS-4735
Project: Mesos
Issue Type: Improvement
Components: fetcher
Affects Versions: 0.27.0
Reporter: Erik Weathers
Priority: Minor
The {{CommandInfo.URI}} message should allow explicitly choosing the downloaded
file's name, to better mimic functionality present in tools like {{wget}} and
{{curl}}.
This relates to issues when the {{CommandInfo.URI}} is pointing to a URL that
has query parameters at the end of the path, resulting in the downloaded
filename having those elements. This also prevents extracting of such files,
since the extraction logic is simply looking at the file's suffix. See
MESOS-3367, MESOS-1686, and MESOS-1509 for more info. If this issue was fixed,
then I could workaround the other issues not being fixed by modifying my
framework's scheduler to set the target filename.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1509) Use Content-Disposition filename (if available) when downloading HTTP URIs
[
https://issues.apache.org/jira/browse/MESOS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105963#comment-15105963
]
Erik Weathers commented on MESOS-1509:
--
[~bmetzdorf]: have you confirmed that mesos currently disregards the
"Content-Disposition filename"? i.e., it wasn't clear to me from the
Description whether you tested with a web server that included the
Content-Disposition header in the response for
{{http://my.web.server/dynamic/resource.tar.gz?a=b}}
> Use Content-Disposition filename (if available) when downloading HTTP URIs
> --
>
> Key: MESOS-1509
> URL: https://issues.apache.org/jira/browse/MESOS-1509
> Project: Mesos
> Issue Type: Improvement
> Components: slave
>Affects Versions: 0.18.0, 0.19.0, 0.20.0, 0.21.0
> Environment: Linux (but should be irrelevant)
>Reporter: Bjoern Metzdorf
>Priority: Minor
>
> Currently the slave stores downloaded HTTP URIs in filenames that are made up
> from the part after the last "/" in the URI (in src/launcher/fetcher.cpp:122):
> {code}
> path = path::join(directory, path.substr(path.find_last_of("/") + 1));
> {code}
> The problem is that the query string is included in the filename and a URI
> like {{http://my.web.server/dynamic/resource.tar.gz?a=b}} results in a
> downloaded file named {{resource.tar.gz?a=b}}.
> The curl maintainers faced the same problem and added this:
> {code}
> -J, --remote-header-name
> (HTTP) This option tells the -O, --remote-name option to use
> the server-specified Content-Disposition filename instead of extracting a
> filename from the URL.
> {code}
> Maybe Mesos could do the same if a Content-Disposition header exists.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3367) Mesos fetcher does not extract archives for URI with parameters
[
https://issues.apache.org/jira/browse/MESOS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105988#comment-15105988
]
Erik Weathers commented on MESOS-3367:
--
Thanks for the attention on this issue [~haosd...@gmail.com] & [~bernd-mesos]!
It's not clear to me how the patches that [~haosd...@gmail.com] posted would
help solve the issue.
Also, the lack of this feature prevents using mesos's fetcher to grab tarballs
from WebHDFS, since when you download a file over WebHDFS the resultant URL
includes a suffix like
{{?op=OPEN=hadoop-namenode=0}}, and with an HTTP
Content-Type of {{application/octet-stream}}. Luckily the Linux {{file}}
command reveals the true nature of the files:
{code}
% wget
http://hadoop-namenode:50070/webhdfs/v1/user/erikdw/storm-mesos-0.9.6.tgz?op=OPEN
% mv
'storm-mesos-0.9.6.tgz?op=OPEN=hadoop-namenode=0'
foo.tar.gz
% file foo.tar.gz
foo.tar.gz: gzip compressed data, from Unix, last modified: Mon Jan 18 22:30:57
2016
% unzip foo.tar.gz
% file foo.tar
foo.tar: POSIX tar archive (GNU)
{code}
NOTE: Unfortunately, {{gzip}} is kind of simplistic and ["cowardly refuses" (a
la
tar)|https://github.com/kikitux/tar/blob/5e2a1d5b3801d016f51b3f4c476d275a6adff5d7/src/tar.c#L2891]
to uncompress a file with an unexpected suffix, hence my renaming above.
Luckily {{tar}} is more robust and can directly extract such a file without
renaming. But... unfortunately, {{tar}} fails if such a "gzip compressed data"
file is not *also* a tar file. Sigh.
> Mesos fetcher does not extract archives for URI with parameters
> ---
>
> Key: MESOS-3367
> URL: https://issues.apache.org/jira/browse/MESOS-3367
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Affects Versions: 0.22.1, 0.23.0
> Environment: DCOS 1.1
>Reporter: Renat Zubairov
>Assignee: haosdent
>Priority: Minor
> Labels: mesosphere
>
> I'm deploying using marathon applications with sources served from S3. I'm
> using a signed URL to give only temporary access to the S3 resources, so URL
> of the resource have some query parameters.
> So URI is 'https://foo.com/file.tgz?hasi' and fetcher stores it in the file
> with the name 'file.tgz?hasi', then it thinks that extension 'hasi' is not
> tgz hence extraction is skipped, despite the fact that MIME Type of the HTTP
> resource is 'application/x-tar'.
> Workaround - add additional parameter like '=.tgz'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3367) Mesos fetcher does not extract archives for URI with parameters
[
https://issues.apache.org/jira/browse/MESOS-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106016#comment-15106016
]
Erik Weathers commented on MESOS-3367:
--
Just found another ticket (MESOS-1686) which proposes just dropping the query
params in the resultant local filename.
Also, it might be valuable to allow the crafter of the {{CommandInfo.URI}} to
explicitly state the desired local filename. I can file a ticket for that if
it's seen as a reasonable idea.
> Mesos fetcher does not extract archives for URI with parameters
> ---
>
> Key: MESOS-3367
> URL: https://issues.apache.org/jira/browse/MESOS-3367
> Project: Mesos
> Issue Type: Improvement
> Components: fetcher
>Affects Versions: 0.22.1, 0.23.0
> Environment: DCOS 1.1
>Reporter: Renat Zubairov
>Assignee: haosdent
>Priority: Minor
> Labels: mesosphere
>
> I'm deploying using marathon applications with sources served from S3. I'm
> using a signed URL to give only temporary access to the S3 resources, so URL
> of the resource have some query parameters.
> So URI is 'https://foo.com/file.tgz?hasi' and fetcher stores it in the file
> with the name 'file.tgz?hasi', then it thinks that extension 'hasi' is not
> tgz hence extraction is skipped, despite the fact that MIME Type of the HTTP
> resource is 'application/x-tar'.
> Workaround - add additional parameter like '=.tgz'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1478) Replace Master/Slave terminology
[ https://issues.apache.org/jira/browse/MESOS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097171#comment-15097171 ] Erik Weathers commented on MESOS-1478: -- [~adam-mesos]: Thank you for the response, that makes sense. It's confusing that the name "slave" continues to exist all over (code, docs, emails, etc.), while in-the-know folks are saying "agent", but I suppose there is no easy way around that. > Replace Master/Slave terminology > > > Key: MESOS-1478 > URL: https://issues.apache.org/jira/browse/MESOS-1478 > Project: Mesos > Issue Type: Epic >Reporter: Clark Breyman >Assignee: Benjamin Hindman >Priority: Minor > Labels: mesosphere > > Inspired by the comments on this PR: > https://github.com/django/django/pull/2692 > TL;DR - Computers sharing work should be a good thing. Using the language of > human bondage and suffering is inappropriate in this context. It also has the > potential to alienate users and community members. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3341) Introduce Resource Resolution
[ https://issues.apache.org/jira/browse/MESOS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097574#comment-15097574 ] Erik Weathers commented on MESOS-3341: -- [~neilc]: seems unlikely that anyone is carving up CPUs at such a granularity. So the question of how to handle this "resource resolution" comes down to what the best behavior to exhibit in the rare case that it *does* happen. My perspective is that we should go with the least surprising behavior, and changing the requested resources in-transit seems to run counter to that principle. > Introduce Resource Resolution > - > > Key: MESOS-3341 > URL: https://issues.apache.org/jira/browse/MESOS-3341 > Project: Mesos > Issue Type: Improvement >Reporter: Jessica Hartog >Priority: Minor > > After MESOS-1807, Mesos containers require >= 0.01 CPU resources and >= 32MB > Memory resources. In order to simplify accounting, Mesos should introduce > resource resolution. > For example, it is possible to launch a task with 1.1 CPU (as it > exceeds the minimum number of CPUs and is therefore considered valid). The > fractional component of this task does not benefit the running process, and > can introduce floating point errors when Mesos is accounting its offers > (which we have already seen happening in MESOS-1867 and MESOS-2635). A > solution to this could be disallowing tasks with finer granularity than the > required resolution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1478) Replace Master/Slave terminology
[ https://issues.apache.org/jira/browse/MESOS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998219#comment-14998219 ] Erik Weathers commented on MESOS-1478: -- If I may ask, can someone please explain where the discussion & conclusion about the choice of the new name happened? I saw an email chain about *whether* to do the rename (which was inconclusive), and then when I attended MesosCon 2015 in Seattle, it was announced "from on high" that the new name was "agent". Was this discussed in some ad hoc informal forum? Decided internally to Mesosphere? > Replace Master/Slave terminology > > > Key: MESOS-1478 > URL: https://issues.apache.org/jira/browse/MESOS-1478 > Project: Mesos > Issue Type: Epic >Reporter: Clark Breyman >Assignee: Benjamin Hindman >Priority: Minor > Labels: mesosphere > > Inspired by the comments on this PR: > https://github.com/django/django/pull/2692 > TL;DR - Computers sharing work should be a good thing. Using the language of > human bondage and suffering is inappropriate in this context. It also has the > potential to alienate users and community members. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-1478) Replace Master/Slave terminology
[ https://issues.apache.org/jira/browse/MESOS-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998219#comment-14998219 ] Erik Weathers edited comment on MESOS-1478 at 11/10/15 8:24 AM: If I may ask, can someone please explain where the discussion & conclusion about the choice of the new name for "slave" happened? I saw an email chain about *whether* to do the rename (which was inconclusive), and then when I attended MesosCon 2015 in Seattle, it was announced "from on high" that the new name was "agent". Was this discussed in some ad hoc informal forum? Decided internally to Mesosphere? was (Author: erikdw): If I may ask, can someone please explain where the discussion & conclusion about the choice of the new name happened? I saw an email chain about *whether* to do the rename (which was inconclusive), and then when I attended MesosCon 2015 in Seattle, it was announced "from on high" that the new name was "agent". Was this discussed in some ad hoc informal forum? Decided internally to Mesosphere? > Replace Master/Slave terminology > > > Key: MESOS-1478 > URL: https://issues.apache.org/jira/browse/MESOS-1478 > Project: Mesos > Issue Type: Epic >Reporter: Clark Breyman >Assignee: Benjamin Hindman >Priority: Minor > Labels: mesosphere > > Inspired by the comments on this PR: > https://github.com/django/django/pull/2692 > TL;DR - Computers sharing work should be a good thing. Using the language of > human bondage and suffering is inappropriate in this context. It also has the > potential to alienate users and community members. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-2743) Include ExecutorInfos for custom executors in master/state.json
[
https://issues.apache.org/jira/browse/MESOS-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14735848#comment-14735848
]
Erik Weathers commented on MESOS-2743:
--
hey [~adam-mesos]: can you please provide a brief explanation about why "only
custom executors are known to the master"?
Prior to that statement, I *was* hopeful that the fix in this ticket
(MESOS-2743) would also fix the following webui behavior I observed in
mesos-0.22.1: when I drill into the Executor-view in the UI, I see a blank
"Executor Name:" value, despite the {{executor_name}} appearing in
{{http://WORKER_HOST_IP:5051/monitor/statistics.json}}
* FYI, Executor-view UI URL (in mesos-0.22.1):
**
{{http://MASTER_IP:5050/slaves/SLAVE_ID/frameworks/FRAMEWORK_ID/executors/EXECUTOR_ID}}
I noted that the {{master/state.json}} endpoint that I believe drives the webui
doesn't include the {{executor_name}} (at least in mesos-0.22.1) and thus this
ticket (MESOS-2743) seemed like it might fix the problem I observed, since the
executor name *is* in {{ExecutorInfo}}. But your statement about "custom
executors" gave me pause. So I'm wondering if I need to file a bug for the
above behavior instead.
Thanks!
> Include ExecutorInfos for custom executors in master/state.json
> ---
>
> Key: MESOS-2743
> URL: https://issues.apache.org/jira/browse/MESOS-2743
> Project: Mesos
> Issue Type: Improvement
> Components: json api
>Reporter: Adam B
>Assignee: haosdent
> Labels: mesosphere
> Fix For: 0.23.0
>
>
> The slave/state.json already reports executorInfos:
> https://github.com/apache/mesos/blob/0.22.1/src/slave/http.cpp#L215-219
> Would be great to see this in the master/state.json as well, so external
> tools don't have to query each slave to find out executor resources, sandbox
> directories, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644867#comment-14644867 ] Erik Weathers commented on MESOS-1807: -- [~vi...@twitter.com]: without the web UI fix then to me its unacceptable to *use* fractional CPUs at all. So for storm-on-mesos I'm using 0 as the executor CPUs (because the mesos-supervisor does almost nothing, 0 is not entirely unreasonable). When the behavior change requested in this ticket is enacted (changing from warning to error) then I will either be stuck on the previous version of mesos, or I will be forced to have an ugly UI. Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Attachments: Screenshot 2015-07-28 14.40.35.png Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644829#comment-14644829 ] Erik Weathers commented on MESOS-1807: -- In mesos-0.20.1 the mesos UI got completely broken if you had fractional CPUs assigned to executors/tasks: * https://issues.apache.org/jira/secure/attachment/12747600/Screenshot%202015-07-28%2014.40.35.png) I would consider that bug a must fix before disallowing 0 CPUs for executors. Maybe it's already fixed? Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Attachments: Screenshot 2015-07-28 14.40.35.png Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644829#comment-14644829 ] Erik Weathers edited comment on MESOS-1807 at 7/28/15 6:42 PM: --- In mesos-0.20.1 the mesos UI got completely broken if you had fractional CPUs assigned to executors/tasks: * https://issues.apache.org/jira/secure/attachment/12747600/Screenshot%202015-07-28%2014.40.35.png I would consider that bug a must fix before disallowing 0 CPUs for executors. Maybe it's already fixed? was (Author: erikdw): In mesos-0.20.1 the mesos UI got completely broken if you had fractional CPUs assigned to executors/tasks: * https://issues.apache.org/jira/secure/attachment/12747600/Screenshot%202015-07-28%2014.40.35.png) I would consider that bug a must fix before disallowing 0 CPUs for executors. Maybe it's already fixed? Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Attachments: Screenshot 2015-07-28 14.40.35.png Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Weathers updated MESOS-1807: - Attachment: Screenshot 2015-07-28 14.40.35.png Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Attachments: Screenshot 2015-07-28 14.40.35.png Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-1807) Disallow executors with cpu only or memory only resources
[ https://issues.apache.org/jira/browse/MESOS-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644867#comment-14644867 ] Erik Weathers edited comment on MESOS-1807 at 7/28/15 8:27 PM: --- [~vi...@twitter.com]: without the web UI fix then to me its unacceptable to *use* fractional CPUs at all. So for storm-on-mesos I'm using 0 as the executor CPUs (because the mesos-supervisor does almost nothing, 0 is not entirely unreasonable). When the behavior change requested in this ticket is enacted (changing from warning to error) then I will either be stuck on the previous version of mesos, or I will be forced to have an ugly UI. My colleague found a couple of bugs that are related to this issue I'm mentioning: * MESOS-2635 * MESOS-1867 was (Author: erikdw): [~vi...@twitter.com]: without the web UI fix then to me its unacceptable to *use* fractional CPUs at all. So for storm-on-mesos I'm using 0 as the executor CPUs (because the mesos-supervisor does almost nothing, 0 is not entirely unreasonable). When the behavior change requested in this ticket is enacted (changing from warning to error) then I will either be stuck on the previous version of mesos, or I will be forced to have an ugly UI. Disallow executors with cpu only or memory only resources - Key: MESOS-1807 URL: https://issues.apache.org/jira/browse/MESOS-1807 Project: Mesos Issue Type: Improvement Reporter: Vinod Kone Labels: newbie Attachments: Screenshot 2015-07-28 14.40.35.png Currently master allows executors to be launched with either only cpus or only memory but we shouldn't allow that. This is because executor is an actual unix process that is launched by the slave. If an executor doesn't specify cpus, what should do the cpu limits be for that executor when there are no tasks running on it? If no cpu limits are set then it might starve other executors/tasks on the slave violating isolation guarantees. Same goes with memory. Moreover, the current containerizer/isolator code will throw failures when using such an executor, e.g., when the last task on the executor finishes and Containerizer::update() is called with 0 cpus or 0 mem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
