[jira] [Commented] (MESOS-9715) Support specifying output file name for curl fetcher plugin
[ https://issues.apache.org/jira/browse/MESOS-9715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815900#comment-16815900 ] Qian Zhang commented on MESOS-9715: --- commit 27b238dc44527cf30815d53579c928052f35add8 Author: Qian Zhang Date: Wed Apr 10 11:00:48 2019 +0800 Added an optional parameter `outputFileName` to the fetcher interface. Review: [https://reviews.apache.org/r/70443] commit b4531d868ffaad9da4b5d96970fbb8441952f620 Author: Qian Zhang Date: Wed Apr 10 14:14:23 2019 +0800 Added an parameter `outputFileName` to the fetcher plugin interface. Review: [https://reviews.apache.org/r/70444] commit c8542db7e9ca149bee5a2c152a838069d6da3022 Author: Qian Zhang Date: Wed Apr 10 14:20:33 2019 +0800 Allowed caller to specify output file name for curl fetcher plugin. Review: [https://reviews.apache.org/r/70445] commit adb9b809898bf612a78c1ea0ba0b9230c3c4a992 Author: Qian Zhang Date: Wed Apr 10 14:46:22 2019 +0800 Added a test `CurlFetcherPluginTest.CURL_ValidUriWithOutputFileName`. Review: https://reviews.apache.org/r/70446 > Support specifying output file name for curl fetcher plugin > --- > > Key: MESOS-9715 > URL: https://issues.apache.org/jira/browse/MESOS-9715 > Project: Mesos > Issue Type: Task >Reporter: Qian Zhang >Assignee: Qian Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9728) Update bundled libevent to 2.1.8 for autotools builds - or unbundle it.
Till Toenshoff created MESOS-9728: - Summary: Update bundled libevent to 2.1.8 for autotools builds - or unbundle it. Key: MESOS-9728 URL: https://issues.apache.org/jira/browse/MESOS-9728 Project: Mesos Issue Type: Task Components: build Affects Versions: 1.8.0 Reporter: Till Toenshoff We should update the bundled libevent to 2.1.8-stable. Alternatively, we might even consider unbundling given that we have no strict version or patch constraints on that dependency. Anything installed by an up-to-date package manager should be fine. Changes since from 2.0.22 to 2.1.8: https://github.com/libevent/libevent/compare/c51b159cff9f5e86696f5b9a4c6f517276056258...e7ff4ef2b4fc950a765008c18e74281cdb5e7668 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9123) Add metric role consumed quota.
[ https://issues.apache.org/jira/browse/MESOS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kone reassigned MESOS-9123: - Assignee: Meng Zhu (was: Till Toenshoff) > Add metric role consumed quota. > --- > > Key: MESOS-9123 > URL: https://issues.apache.org/jira/browse/MESOS-9123 > Project: Mesos > Issue Type: Improvement > Components: allocation >Reporter: Meng Zhu >Assignee: Meng Zhu >Priority: Critical > Labels: allocator, mesosphere, metrics, resource-management > > Currently, quota related metrics exposes quota guarantee and allocated quota. > We should expose "consumed" which is allocated quota plus unallocated > reservations. We already have this info in the allocator as > `consumedQuotaScalarQuantities`, just needs to expose it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9697) Release RPMs are not uploaded to bintray
[ https://issues.apache.org/jira/browse/MESOS-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815449#comment-16815449 ] Till Toenshoff commented on MESOS-9697: --- That second job is definitely hardwired to Kapil's accounts - among them, his Mesos mirror. Gave it a spin with a version "1.6.0" tag. https://builds.apache.org/job/Mesos/job/Packaging/job/CentosRPMs/23/console > Release RPMs are not uploaded to bintray > > > Key: MESOS-9697 > URL: https://issues.apache.org/jira/browse/MESOS-9697 > Project: Mesos > Issue Type: Bug > Components: build >Affects Versions: 1.6.2, 1.7.2, 1.8.0 >Reporter: Benjamin Bannier >Assignee: Benno Evers >Priority: Critical > Labels: foundations, integration, jenkins, packaging, rpm > > While we currently build release RPMs, e.g., > [https://builds.apache.org/view/M-R/view/Mesos/job/Packaging/job/CentOS/job/1.7.x/], > these artifacts are not uploaded to bintray. Due to that RPM links on the > downloads page [http://mesos.apache.org/downloads/] are broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9712) StorageLocalResourceProviderTest.CsiPluginRpcMetrics is flaky
[
https://issues.apache.org/jira/browse/MESOS-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815397#comment-16815397
]
Benjamin Bannier commented on MESOS-9712:
-
It is pretty hard to figure out what exactly goes wrong here from the logs
alone, so I instrumented the code with address sanitizer which lead me to
uncover a use-after-free bug in SLRP. I am not sure that fully fixes the issue,
but looks related.
Review: [https://reviews.apache.org/r/70454/]
> StorageLocalResourceProviderTest.CsiPluginRpcMetrics is flaky
> -
>
> Key: MESOS-9712
> URL: https://issues.apache.org/jira/browse/MESOS-9712
> Project: Mesos
> Issue Type: Bug
> Components: storage
>Affects Versions: 1.8.0, 1.9.0
> Environment: Debian 9, Mesos configured with SSL support
>Reporter: Jan Schlicht
>Assignee: Benjamin Bannier
>Priority: Major
> Labels: storage
>
> From an internal CI run:
> {noformat}
> [ RUN ] StorageLocalResourceProviderTest.CsiPluginRpcMetrics
> 06:56:26 I0409 06:56:26.350445 23181 cluster.cpp:176] Creating default
> 'local' authorizer
> 06:56:26 malloc_consolidate(): invalid chunk size
> 06:56:26 *** Aborted at 1554792986 (unix time) try "date -d @1554792986" if
> you are using GNU date ***
> 06:56:26 PC: @ 0x7f1cf4481f3b (unknown)
> 06:56:26 *** SIGABRT (@0x5a8d) received by PID 23181 (TID 0x7f1ce9be8700)
> from PID 23181; stack trace: ***
> 06:56:26 @ 0x7f1cf461b8e0 __GI___pthread_rwlock_rdlock
> 06:56:26 @ 0x7f1cf4481f3b (unknown)
> 06:56:26 @ 0x7f1cf44832f1 (unknown)
> 06:56:26 @ 0x7f1cf44c4867 (unknown)
> 06:56:26 @ 0x7f1cf44cae0a (unknown)
> 06:56:26 @ 0x7f1cf44cb10e (unknown)
> 06:56:26 @ 0x7f1cf44cddad (unknown)
> 06:56:26 @ 0x7f1cf44cf7dd (unknown)
> 06:56:26 @ 0x7f1cf4a647a8 (unknown)
> 06:56:26 @ 0x7f1cf88d0805 google::LogMessage::Init()
> 06:56:26 @ 0x7f1cf88d10ac google::LogMessage::LogMessage()
> 06:56:26 @ 0x7f1cf752a46a
> mesos::internal::master::Master::initialize()
> 06:56:26 @ 0x7f1cf882bd72 process::ProcessManager::resume()
> 06:56:26 @ 0x7f1cf88303c6
> _ZNSt6thread11_State_implISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> 06:56:26 @ 0x7f1cf4a8ee6f (unknown)
> 06:56:26 @ 0x7f1cf4610f2a (unknown)
> 06:56:26 @ 0x7f1cf4543edf (unknown)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-2842) Master crashes when framework changes principal on re-registration
[ https://issues.apache.org/jira/browse/MESOS-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815287#comment-16815287 ] Andrei Sekretenko commented on MESOS-2842: -- Reviews: https://reviews.apache.org/r/70408/ https://reviews.apache.org/r/70377/ https://reviews.apache.org/r/70379 > Master crashes when framework changes principal on re-registration > -- > > Key: MESOS-2842 > URL: https://issues.apache.org/jira/browse/MESOS-2842 > Project: Mesos > Issue Type: Bug >Reporter: Vinod Kone >Assignee: Andrei Sekretenko >Priority: Critical > Labels: foundations, security > > The master should be updated to avoid crashing when a framework re-registers > with a different principal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9712) StorageLocalResourceProviderTest.CsiPluginRpcMetrics is flaky
[
https://issues.apache.org/jira/browse/MESOS-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Bannier reassigned MESOS-9712:
---
Assignee: Benjamin Bannier
> StorageLocalResourceProviderTest.CsiPluginRpcMetrics is flaky
> -
>
> Key: MESOS-9712
> URL: https://issues.apache.org/jira/browse/MESOS-9712
> Project: Mesos
> Issue Type: Bug
> Components: storage
> Environment: Debian 9, Mesos configured with SSL support
>Reporter: Jan Schlicht
>Assignee: Benjamin Bannier
>Priority: Major
>
> From an internal CI run:
> {noformat}
> [ RUN ] StorageLocalResourceProviderTest.CsiPluginRpcMetrics
> 06:56:26 I0409 06:56:26.350445 23181 cluster.cpp:176] Creating default
> 'local' authorizer
> 06:56:26 malloc_consolidate(): invalid chunk size
> 06:56:26 *** Aborted at 1554792986 (unix time) try "date -d @1554792986" if
> you are using GNU date ***
> 06:56:26 PC: @ 0x7f1cf4481f3b (unknown)
> 06:56:26 *** SIGABRT (@0x5a8d) received by PID 23181 (TID 0x7f1ce9be8700)
> from PID 23181; stack trace: ***
> 06:56:26 @ 0x7f1cf461b8e0 __GI___pthread_rwlock_rdlock
> 06:56:26 @ 0x7f1cf4481f3b (unknown)
> 06:56:26 @ 0x7f1cf44832f1 (unknown)
> 06:56:26 @ 0x7f1cf44c4867 (unknown)
> 06:56:26 @ 0x7f1cf44cae0a (unknown)
> 06:56:26 @ 0x7f1cf44cb10e (unknown)
> 06:56:26 @ 0x7f1cf44cddad (unknown)
> 06:56:26 @ 0x7f1cf44cf7dd (unknown)
> 06:56:26 @ 0x7f1cf4a647a8 (unknown)
> 06:56:26 @ 0x7f1cf88d0805 google::LogMessage::Init()
> 06:56:26 @ 0x7f1cf88d10ac google::LogMessage::LogMessage()
> 06:56:26 @ 0x7f1cf752a46a
> mesos::internal::master::Master::initialize()
> 06:56:26 @ 0x7f1cf882bd72 process::ProcessManager::resume()
> 06:56:26 @ 0x7f1cf88303c6
> _ZNSt6thread11_State_implISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> 06:56:26 @ 0x7f1cf4a8ee6f (unknown)
> 06:56:26 @ 0x7f1cf4610f2a (unknown)
> 06:56:26 @ 0x7f1cf4543edf (unknown)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (MESOS-9697) Release RPMs are not uploaded to bintray
[ https://issues.apache.org/jira/browse/MESOS-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814545#comment-16814545 ] Benno Evers edited comment on MESOS-9697 at 4/11/19 9:12 AM: - After some investigation, here's my current understanding of the situation: * The ASF Jenkins is successfully running the `Mesos/Packaging/CentOS` job ( https://builds.apache.org/view/M-R/view/Mesos/job/Packaging/job/CentOS/ ) to a branch that contains the file `support/jenkins/Jenkinsfile-packaging-centos`, i.e. currently branches 1.7.x, 1.8.x and master. This jenkinsfile creates rpm packages for centos 6 and 7 as artifacts (using the script `support/packaging/centos/build-rpm-docker.sh`), but does not do anything with them, i.e. there is no connection to bintray. I don't know if there is any public download for the generated artifacts. * The is another job `Mesos/Packaging/CentosRPMs` (https://builds.apache.org/job/Mesos/job/Packaging/job/CentosRPMs) defined in the ASF Jenkins that is not run automatically. For its setup, its using the file `support/packaging/Jenkinsfile` from branch `bintray` on `http://github.com/karya0/mesos.git`. It is taking parameters `MESOS_RELEASE` and `MESOS_TAG` and will build centos 6/7 rpm packages for that release (I still don't understand where exactly it's taking the source code from) and afterwards upload them to bintray using credentials "karya_bintray_credentials". It was last run by [~karya] on Feb 8, 2018 to produce Mesos 1.5.0 packages. So it looks like this might not actually be broken, but rather just release managers not being aware that they are supposed to manually run this Jenkins job. I'd like to test that theory by triggering a 1.7.0 build of the latter job, but I don't seem to have permissions to do that on the ASF Jenkins. was (Author: bennoe): After some investigation, here's my current understanding of the situation: * The ASF Jenkins is successfully running the `Mesos/Packaging/CentOS` job ( https://builds.apache.org/view/M-R/view/Mesos/job/Packaging/job/CentOS/ ) to a branch that contains the file `support/jenkins/Jenkinsfile-packaging-centos`, i.e. currently branches 1.7.x, 1.8.x and master. This jenkinsfile creates rpm packages for centos 6 and 7 as artifacts (using the script `support/packaging/centos/build-rpm-docker.sh`), but does not do anything with them, i.e. there is no connection to bintray. I don't know if there is any public download for the generated artifacts. * The is another job `Mesos/Packaging/CentosRPMs` (https://builds.apache.org/job/Mesos/job/Packaging/job/CentosRPMs) defined in the ASF Jenkins that is not run manually. For its setup, its using the file `support/packaging/Jenkinsfile` from branch `bintray` on `http://github.com/karya0/mesos.git`. It is taking parameters `MESOS_RELEASE` and `MESOS_TAG` and will build centos 6/7 rpm packages for that release (I still don't understand where exactly it's taking the source code from) and afterwards upload them to bintray using credentials "karya_bintray_credentials". It was last run by [~karya] on Feb 8, 2018 to produce Mesos 1.5.0 packages. So it looks like this might not actually be broken, but rather just release managers not being aware that they are supposed to manually run this Jenkins job. I'd like to test that theory by triggering a 1.7.0 build of the latter job, but I don't seem to have permissions to do that on the ASF Jenkins. > Release RPMs are not uploaded to bintray > > > Key: MESOS-9697 > URL: https://issues.apache.org/jira/browse/MESOS-9697 > Project: Mesos > Issue Type: Bug > Components: build >Affects Versions: 1.6.2, 1.7.2, 1.8.0 >Reporter: Benjamin Bannier >Assignee: Benno Evers >Priority: Critical > Labels: foundations, integration, jenkins, packaging, rpm > > While we currently build release RPMs, e.g., > [https://builds.apache.org/view/M-R/view/Mesos/job/Packaging/job/CentOS/job/1.7.x/], > these artifacts are not uploaded to bintray. Due to that RPM links on the > downloads page [http://mesos.apache.org/downloads/] are broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9699) Pull in glog 0.4.0
[ https://issues.apache.org/jira/browse/MESOS-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815236#comment-16815236 ] Andrei Sekretenko commented on MESOS-9699: -- Review: https://reviews.apache.org/r/70388/ > Pull in glog 0.4.0 > -- > > Key: MESOS-9699 > URL: https://issues.apache.org/jira/browse/MESOS-9699 > Project: Mesos > Issue Type: Task >Reporter: Andrei Sekretenko >Assignee: Andrei Sekretenko >Priority: Major > Labels: foundations, mesosphere > > There are at the very least two reasons to update. > - We will be able to get rid of patches for Windows build > https://issues.apache.org/jira/browse/MESOS-3394 and a huge number of other > patches, and will switch cmake build to building glog with cmake on Linux/BSD > (now it uses autotools under cmake!) > - The microseconds patch introduced in > https://issues.apache.org/jira/browse/MESOS-9687 will still remain, but at > least there will be no _backported_ microseconds patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9594) Test `StorageLocalResourceProviderTest.RetryRpcWithExponentialBackoff` is flaky.
[
https://issues.apache.org/jira/browse/MESOS-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815177#comment-16815177
]
Jan Schlicht commented on MESOS-9594:
-
While trying to reproduce this locally, running
{noformat}
stress-ng --cpu=100 --io 20 --vm 20 --fork 100 --timeout 3600s &
GLOG_v=1 src/mesos-tests --verbose
--gtest_filter=*RetryRpcWithExponentialBackoff --gtest_repeat=-1
--gtest_break_on_failure
{noformat}
this crashes in a similar manner as reported in MESOS-9712. Log:
[^RetryRpcWithExponentialBackoff-segfault.txt]
> Test `StorageLocalResourceProviderTest.RetryRpcWithExponentialBackoff` is
> flaky.
>
>
> Key: MESOS-9594
> URL: https://issues.apache.org/jira/browse/MESOS-9594
> Project: Mesos
> Issue Type: Bug
> Components: storage, test
>Reporter: Chun-Hung Hsiao
>Assignee: Jan Schlicht
>Priority: Major
> Labels: flaky-test, mesosphere, storage
> Attachments: RetryRpcWithExponentialBackoff-badrun.txt,
> RetryRpcWithExponentialBackoff-segfault.txt
>
>
> Observed on ASF CI:
> {noformat}
> /tmp/SRC/src/tests/storage_local_resource_provider_tests.cpp:5027
> Failed to wait 1mins for offers
> {noformat}
> Full log: [^RetryRpcWithExponentialBackoff-badrun.txt]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (MESOS-9703) Support docker manifest v2s2 external urls.
[ https://issues.apache.org/jira/browse/MESOS-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815145#comment-16815145 ] Qian Zhang edited comment on MESOS-9703 at 4/11/19 7:51 AM: Currently in Docker the external URLs is only supported on Windows but not other OS, see [this PR|https://github.com/moby/moby/pull/22866] for details. And I see there is [another PR|https://github.com/moby/moby/pull/23014] tried to support external URLs for other OS, but that PR was closed somehow. In the [`open` method in pull_v2_windows.go|https://github.com/moby/moby/blob/67fdf574d5acd6ddccb6ece0ffe0ace1c1608712/distribution/pull_v2_windows.go#L25:L57], we can see Docker tries layer's digest first, but if it is not accessible from registry, then it tries the external URLs one by one until an external URL is accessible. But in the [`open` method in pull_v2_unix.go|https://github.com/moby/moby/blob/05bd04350b8348b3c3bbe3156420257313e4e804/distribution/pull_v2_unix.go#L16:L19], only layer's digest will be tried. So external URLs is only used on Window but ignored on Unix-like OS. And in the [`Descriptor` method in pull_v2_windows.go|https://github.com/moby/moby/blob/67fdf574d5acd6ddccb6ece0ffe0ace1c1608712/distribution/pull_v2_windows.go#L18:L23], we can see the external URLs will only be used when the layer's media type is `application/vnd.docker.image.rootfs.foreign.diff.tar.gzip` otherwise it will be just ignored. was (Author: qianzhang): Currently in Docker the external URLs is only supported on Windows but not other OS, see [this PR|https://github.com/moby/moby/pull/22866] for details. And I see there is [another PR|https://github.com/moby/moby/pull/23014] tried to support external URLs for other OS, but that PR was closed somehow. In the [`open` method of pull_v2_windows.go|https://github.com/moby/moby/blob/67fdf574d5acd6ddccb6ece0ffe0ace1c1608712/distribution/pull_v2_windows.go#L25:L57], we can see Docker tries layer's digest first, but if it is not accessible from registry, then it tries the external URLs one by one until an external URL is accessible. But in the [`open` method of pull_v2_unix.go|https://github.com/moby/moby/blob/05bd04350b8348b3c3bbe3156420257313e4e804/distribution/pull_v2_unix.go#L16:L19], only layer's digest will be tried. So external URLs is only used on Window but ignored on Unix-like OS. > Support docker manifest v2s2 external urls. > --- > > Key: MESOS-9703 > URL: https://issues.apache.org/jira/browse/MESOS-9703 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Gilbert Song >Assignee: Qian Zhang >Priority: Major > Labels: containerization > > docker manifest v2s2 spec define external URLs. Some windows image rely on > those urls to download some private layers from microsoft server. > Some refactoring may be needed to get rid of the current external urls > support because the uri fetcher has to parse the manifest when pulling every > layer (see the patches of MESOS-9159 for details). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9703) Support docker manifest v2s2 external urls.
[ https://issues.apache.org/jira/browse/MESOS-9703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815145#comment-16815145 ] Qian Zhang commented on MESOS-9703: --- Currently in Docker the external URLs is only supported on Windows but not other OS, see [this PR|https://github.com/moby/moby/pull/22866] for details. And I see there is [another PR|https://github.com/moby/moby/pull/23014] tried to support external URLs for other OS, but that PR was closed somehow. In the [`open` method of pull_v2_windows.go|https://github.com/moby/moby/blob/67fdf574d5acd6ddccb6ece0ffe0ace1c1608712/distribution/pull_v2_windows.go#L25:L57], we can see Docker tries layer's digest first, but if it is not accessible from registry, then it tries the external URLs one by one until an external URL is accessible. But in the [`open` method of pull_v2_unix.go|https://github.com/moby/moby/blob/05bd04350b8348b3c3bbe3156420257313e4e804/distribution/pull_v2_unix.go#L16:L19], only layer's digest will be tried. So external URLs is only used on Window but ignored on Unix-like OS. > Support docker manifest v2s2 external urls. > --- > > Key: MESOS-9703 > URL: https://issues.apache.org/jira/browse/MESOS-9703 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: Gilbert Song >Assignee: Qian Zhang >Priority: Major > Labels: containerization > > docker manifest v2s2 spec define external URLs. Some windows image rely on > those urls to download some private layers from microsoft server. > Some refactoring may be needed to get rid of the current external urls > support because the uri fetcher has to parse the manifest when pulling every > layer (see the patches of MESOS-9159 for details). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
