[jira] [Commented] (MESOS-9799) Adopt container file operations in secrets volumes.
[ https://issues.apache.org/jira/browse/MESOS-9799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850489#comment-16850489 ] James Peach commented on MESOS-9799: | [r/70741|https://reviews.apache.org/r/70741] | Adopted container file operations for secrets volumes. | > Adopt container file operations in secrets volumes. > --- > > Key: MESOS-9799 > URL: https://issues.apache.org/jira/browse/MESOS-9799 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: James Peach >Assignee: James Peach >Priority: Major > > Adopt containerized file operations in the secrets volume isolator so that it > doesn't have to use pre-exec commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9800) libarchive cannot extract tarfile due to UTF-8 encoding issues
[
https://issues.apache.org/jira/browse/MESOS-9800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850170#comment-16850170
]
Joseph Wu commented on MESOS-9800:
--
I believe this error appears when the locale of the program (Mesos agent) is
set to some non-UTF locale, such as the default POSIX locale. The addition of
the following line _might_ be enough to fix it.
{code}
diff --git a/3rdparty/stout/include/stout/archiver.hpp
b/3rdparty/stout/include/stout/archiver.hpp
index 551e644a3..706ba5282 100644
--- a/3rdparty/stout/include/stout/archiver.hpp
+++ b/3rdparty/stout/include/stout/archiver.hpp
@@ -54,6 +54,11 @@ inline Try extract(
archive_read_support_format_all(reader.get());
archive_read_support_filter_all(reader.get());
+ // Prevent Libarchive from trying to convert filenames to the locale-default
+ // character encoding. This conversion sometimes fails, for example when
+ // reading UTF-8 characters in a standard "C" locale (POSIX default).
+ archive_read_set_options(reader.get(), "hdrcharset=BINARY");
+
std::unique_ptr> writer(
archive_write_disk_new(),
[](struct archive* p) {
{code}
Could you attach an example archive that fails to extract and the locale of the
machine running your agent (i.e. the output of the {{locale}} command)? That
should give me a better idea of what case is failing.
> libarchive cannot extract tarfile due to UTF-8 encoding issues
> --
>
> Key: MESOS-9800
> URL: https://issues.apache.org/jira/browse/MESOS-9800
> Project: Mesos
> Issue Type: Bug
> Components: fetcher
>Affects Versions: 1.7.2
> Environment: Mesos 1.7.2 and Marathon 1.4.3 running on top of Ubuntu
> 16.04.
>Reporter: Felipe Alfaro Solana
>Priority: Major
>
> Starting with Mesos 1.7, the following change has been introduced:
> * [MESOS-8064] - Mesos now requires libarchive to programmatically decode
> .zip, .tar, .gzip, and other common file compression schemes. Version 3.3.2
> is bundled in Mesos.
> However, this version of libarchive which is used by the fetcher component in
> Mesos has problems in dealing with archive files (.tar and .zip) which
> contain UTF-8 characters. We run Marahton on top of Mesos, and one of our
> Marathon application relies on a .tar file which contains symlinks whose
> target contains certain UTF-8 characters (Turkish) or the symlink name itself
> contains UTF-8 characters. Mesos fetcher is unable to extract the archive and
> fails with the following error:
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
> 10:47:30.791250 6136 fetcher.cpp:613] EXIT with status 1: Failed to fetch
> '/tmp/certificates.tar.gz': Failed to extract archive
> '/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0/certificates.tar.gz'
> to
> '/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0':
> Failed to read archive header: Linkname can't be converted from UTF-8 to
> current locale.}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]:}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: End
> fetcher log for container 6a6e87e8-5eef-4e8e-8c00-3f081fa187b0}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
> 10:47:30.846695 4343 fetcher.cpp:571] Failed to run mesos-fetcher: Failed to
> fetch all URIs for container '6a6e87e8-5eef-4e8e-8c00-3f081fa187b0': exited
> with status 1}}
> The same Marathon application works fine with Mesos 1.6 which does not use
> libarchive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (MESOS-9801) Use protobuf arenas for v1 API responses.
[
https://issues.apache.org/jira/browse/MESOS-9801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Mahler reassigned MESOS-9801:
--
Assignee: William Mahler
> Use protobuf arenas for v1 API responses.
> -
>
> Key: MESOS-9801
> URL: https://issues.apache.org/jira/browse/MESOS-9801
> Project: Mesos
> Issue Type: Improvement
> Components: agent, master
>Reporter: Benjamin Mahler
>Assignee: William Mahler
>Priority: Major
> Labels: performance
>
> The v1 API response construction is currently slower than the v0 API response
> construction. A primary reason for this is that the v1 API constructs
> intermediate C++ protobuf response objects, which are very expensive in terms
> of memory allocation/deallocation cost. Also involved is the use of
> {{evolve()}} which evolves messages from unversioned protobuf into v1
> protobuf. This also has very high memory allocation / deallocation cost.
> Using arenas for all v1 API response construction will provide a significant
> improvement.
> This ticket currently captures all the aspects of this:
> * Updating {{evolve()}} to use arenas across all v1 API responses.
> * Updating all response construction functions (e.g. {{getState())}}) to use
> arenas.
> * Making this change for both the master and agent.
> This is blocked by MESOS-9755 since we need to upgrade our bundled protobuf
> to have string fields allocated in the arenas.
> We may split out tickets for CHANGELOG purposes if only a portion of this
> lands in 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (MESOS-9801) Use protobuf arenas for v1 API responses.
Benjamin Mahler created MESOS-9801:
--
Summary: Use protobuf arenas for v1 API responses.
Key: MESOS-9801
URL: https://issues.apache.org/jira/browse/MESOS-9801
Project: Mesos
Issue Type: Improvement
Components: agent, master
Reporter: Benjamin Mahler
The v1 API response construction is currently slower than the v0 API response
construction. A primary reason for this is that the v1 API constructs
intermediate C++ protobuf response objects, which are very expensive in terms
of memory allocation/deallocation cost. Also involved is the use of
{{evolve()}} which evolves messages from unversioned protobuf into v1 protobuf.
This also has very high memory allocation / deallocation cost.
Using arenas for all v1 API response construction will provide a significant
improvement.
This ticket currently captures all the aspects of this:
* Updating {{evolve()}} to use arenas across all v1 API responses.
* Updating all response construction functions (e.g. {{getState())}}) to use
arenas.
* Making this change for both the master and agent.
This is blocked by MESOS-9755 since we need to upgrade our bundled protobuf to
have string fields allocated in the arenas.
We may split out tickets for CHANGELOG purposes if only a portion of this lands
in 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9798) How to reduce compile time after had changed/improved source code?
[
https://issues.apache.org/jira/browse/MESOS-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849982#comment-16849982
]
Joseph Wu commented on MESOS-9798:
--
This is an architectural problem with how large is the codebase is, and how the
code is structured. The Mesos agent takes the longest to compile, mostly
because 80% or so of the Mesos source files are compiled into the agent. That
includes {{src/docker/docker.hpp}}.
> How to reduce compile time after had changed/improved source code?
> --
>
> Key: MESOS-9798
> URL: https://issues.apache.org/jira/browse/MESOS-9798
> Project: Mesos
> Issue Type: Improvement
> Components: cmake
>Affects Versions: 1.8.0
> Environment: Linux firework-vm01 4.9.0-9-amd64 #1 SMP Debian
> 4.9.168-1+deb9u2 (2019-05-13) x86_64 GNU/Linux
>Reporter: chatsiri
>Priority: Minor
> Labels: newbie
>
> Hello all,
> I'm have changed variables in src/ directory finished, but compiler
> using long time to finished build steps. How can reduces compile time per
> component or source directory? Such as an simple steps below
> # I'm add new member function to class Docker on docker.hpp. This class
> declares on file at docker directory.
> # Compile source again from build directory. This directory create on the
> base source code directory same src/ , bin/ and include/.
> # Come to build path with
> ## $cd build
> ## $../configure --disable-python --disable-java --enable-debug
> --enable-fast-install
> ## $make
> ## $sudo make install.
> In steps No.3. Compiler used long time compiles source code. How we can
> reduce compile time per source directory that we had changed its?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-9800) libarchive cannot extract tarfile due to UTF-8 encoding issues
[
https://issues.apache.org/jira/browse/MESOS-9800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849924#comment-16849924
]
Andrei Budnik commented on MESOS-9800:
--
Thanks for filing a detailed ticket!
Hope [~kaysoky] might help you with this issue.
> libarchive cannot extract tarfile due to UTF-8 encoding issues
> --
>
> Key: MESOS-9800
> URL: https://issues.apache.org/jira/browse/MESOS-9800
> Project: Mesos
> Issue Type: Bug
> Components: fetcher
>Affects Versions: 1.7.2
> Environment: Mesos 1.7.2 and Marathon 1.4.3 running on top of Ubuntu
> 16.04.
>Reporter: Felipe Alfaro Solana
>Priority: Major
>
> Starting with Mesos 1.7, the following change has been introduced:
> * [MESOS-8064] - Mesos now requires libarchive to programmatically decode
> .zip, .tar, .gzip, and other common file compression schemes. Version 3.3.2
> is bundled in Mesos.
> However, this version of libarchive which is used by the fetcher component in
> Mesos has problems in dealing with archive files (.tar and .zip) which
> contain UTF-8 characters. We run Marahton on top of Mesos, and one of our
> Marathon application relies on a .tar file which contains symlinks whose
> target contains certain UTF-8 characters (Turkish) or the symlink name itself
> contains UTF-8 characters. Mesos fetcher is unable to extract the archive and
> fails with the following error:
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
> 10:47:30.791250 6136 fetcher.cpp:613] EXIT with status 1: Failed to fetch
> '/tmp/certificates.tar.gz': Failed to extract archive
> '/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0/certificates.tar.gz'
> to
> '/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0':
> Failed to read archive header: Linkname can't be converted from UTF-8 to
> current locale.}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]:}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: End
> fetcher log for container 6a6e87e8-5eef-4e8e-8c00-3f081fa187b0}}
> {{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
> 10:47:30.846695 4343 fetcher.cpp:571] Failed to run mesos-fetcher: Failed to
> fetch all URIs for container '6a6e87e8-5eef-4e8e-8c00-3f081fa187b0': exited
> with status 1}}
> The same Marathon application works fine with Mesos 1.6 which does not use
> libarchive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (MESOS-9800) libarchive cannot extract tarfile due to UTF-8 encoding issues
Felipe Alfaro Solana created MESOS-9800:
---
Summary: libarchive cannot extract tarfile due to UTF-8 encoding
issues
Key: MESOS-9800
URL: https://issues.apache.org/jira/browse/MESOS-9800
Project: Mesos
Issue Type: Bug
Components: fetcher
Affects Versions: 1.7.2
Environment: Mesos 1.7.2 and Marathon 1.4.3 running on top of Ubuntu
16.04.
Reporter: Felipe Alfaro Solana
Starting with Mesos 1.7, the following change has been introduced:
* [MESOS-8064] - Mesos now requires libarchive to programmatically decode
.zip, .tar, .gzip, and other common file compression schemes. Version 3.3.2 is
bundled in Mesos.
However, this version of libarchive which is used by the fetcher component in
Mesos has problems in dealing with archive files (.tar and .zip) which contain
UTF-8 characters. We run Marahton on top of Mesos, and one of our Marathon
application relies on a .tar file which contains symlinks whose target contains
certain UTF-8 characters (Turkish) or the symlink name itself contains UTF-8
characters. Mesos fetcher is unable to extract the archive and fails with the
following error:
{{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
10:47:30.791250 6136 fetcher.cpp:613] EXIT with status 1: Failed to fetch
'/tmp/certificates.tar.gz': Failed to extract archive
'/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0/certificates.tar.gz'
to
'/var/mesos/slaves/10c35371-f690-4d40-8b9e-30ffd04405fb-S6/frameworks/ff2993eb-987f-47b0-b3af-fb8b49ab0470-/executors/test-nginx.fe01a0c0-8135-11e9-a160-02427a38aa03/runs/6a6e87e8-5eef-4e8e-8c00-3f081fa187b0':
Failed to read archive header: Linkname can't be converted from UTF-8 to
current locale.}}
{{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]:}}
{{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: End
fetcher log for container 6a6e87e8-5eef-4e8e-8c00-3f081fa187b0}}
{{May 28 10:47:30 t01m01.node.t01.dns.teralytics.net mesos-slave[4319]: E0528
10:47:30.846695 4343 fetcher.cpp:571] Failed to run mesos-fetcher: Failed to
fetch all URIs for container '6a6e87e8-5eef-4e8e-8c00-3f081fa187b0': exited
with status 1}}
The same Marathon application works fine with Mesos 1.6 which does not use
libarchive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (MESOS-9254) Make SLRP be able to update its volumes and storage pools.
[ https://issues.apache.org/jira/browse/MESOS-9254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Orlando Hohmeier reassigned MESOS-9254: --- Assignee: Benjamin Bannier (was: Chun-Hung Hsiao) > Make SLRP be able to update its volumes and storage pools. > -- > > Key: MESOS-9254 > URL: https://issues.apache.org/jira/browse/MESOS-9254 > Project: Mesos > Issue Type: Improvement >Reporter: Chun-Hung Hsiao >Assignee: Benjamin Bannier >Priority: Critical > Labels: mesosphere, mesosphere-dss-post-ga, storage > > We should consider making SLRP update its resources periodically, or adding > an endpoint to trigger that, for the following reasons: > 1. Mesos currently assumes all profiles have disjoint storage pools. This is > because Mesos models each resource independently. However, in practice an > operator can set up, say two profiles, one for linear volumes and one for > raid volumes, and an "LVM" resource provider that can provision both linear > and raid volumes. The correlation between the storage pools of the linear and > raid profiles would reduce one's pool capacity when a volume of the other > type is provisioned. To reflect the actual sizes of correlated storage pools, > we need a way to make SLRP update its resources. > 2. The SLRP now only queries the CSI plugin to report a list of volumes > during startup, so if a new device is added, the operator will have to > restart the agent to trigger another SLRP startup, which is inconvenient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
