Re: How to commit code change to kafka mesos framework to fix known issues?

[tommy xiao]

here? or PR

2016-11-28 20:32 GMT+08:00 Yu Wei :

> Hi Guys,
>
>
> I  want to fix a problem in kafka mesos framework.
>
> https://github.com/mesos/kafka/issues/197
>
>
> How could I do that? Is there any process about this?
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>



-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: MESOS-6233 Allow agents to re-register post a host reboot

[tommy xiao]

agree with james's options.

2016-11-30 0:48 GMT+08:00 James Peach :

>
> > On Nov 28, 2016, at 6:09 PM, Yan Xu  wrote:
> >
> > So one thing that was brought up during offline conversations was that
> if the host reboot is associated with hardware change (e.g., a new memory
> stick):
> >
> >   • Currently: the agent would skip the recovery (and the chance of
> running into incompatible agent info) and register as a new agent.
> >   • With the change: the agent could run into incompatible agent
> info due to resource change and flap indefinitely until the operator
> intervenes.
> >
> > To mitigate this and maintain the current behavior, we can have the
> agent remove `rm -f /meta/slaves/latest` automatically upon
> recovery failure but only after the host has rebooted. This way the agent
> can restart as a new agent without operator intervention.
> >
> > Any thoughts?
>
> I still think you need a mechanism for the master/agent to tell you
> whether it will honor the restart policy. Without this, you have to lock
> the framework to a Mesos version.
>
> An empty RestartPolicy is also problematic since it precludes using
> RestartPolicy in pods. If you later want to restart a task inside a pod but
> not across agent restarts you would have no way to express that.
>
> J




-- 
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: Hook Deprecation: slavePreLaunchDockerEnvironmentDecorator

[Till Toenshoff]

Yes, thanks for these additions Adam - much appreciated.

…and while we are at it, we will also remove the `slavePreLaunchDockerHook` 
which is equally superseded by that new hook.


> On Nov 29, 2016, at 7:46 PM, Adam Bordelon  wrote:
> 
> + dev@, in case some module developers haven't joined the modules@ list yet.
> So this will require code changes (and a recompile) so any module
> implementing the previous hook must now use the new hook, potentially
> updating to set both environments instead of just the one. Such a hook
> compiled against Mesos 1.1 will not work anymore with Mesos 1.2.
> 
> On Tue, Nov 29, 2016 at 10:15 AM, Till Toenshoff  wrote:
> 
>> Hey Folks,
>> 
>> we are removing the slave sided, docker specific hook callback `
>> slavePreLaunchDockerEnvironmentDecorator` and replace it with `
>> slavePreLaunchDockerTaskExecutorDecorator`.
>> 
>> The latter now allows setting two individual environments, one effective
>> for the task and one for the executor. Note that as usual, a custom
>> executor itself may still decide to make (parts of) the executor
>> environment available to its task/s.
>> 
>> We decided to use a protobuf message return signature for that hook,
>> allowing us to add things like e.g. volumes without having to come up with
>> another hook callback — see discussion around https://issues.apache.org/
>> jira/browse/MESOS-6396?focusedCommentId=15654316=com.atlassian.jira.
>> plugin.system.issuetabpanels:comment-tabpanel#comment-15654316.
>> 
>> The changes involved should have no extra side-effects - your
>> functionality is unaffected if you don't use custom hooks.
>> 
>> Please remember that hooks are still considered experimental and may still
>> evolve fast.
>> 
>> 
>> Let me know what you think!
>> 
>> Till

Re: [VOTE] Release Apache Mesos 0.28.3 (rc1)

[Joseph Wu]

AlexR,

Thanks for pointing out those test failures.  As of 0.28, the
LinuxFilesystemIsolatorTests were notoriously flaky on distributions with
"large" root filesystems.  The test would essentially copy the root
filesystem, leading to timeouts in multiple places in the tests.  CentOS 7
was known to have at least twice as much stuff to copy compared to the
other distributions (not sure about Fedora 23).

Looking at your logs (and logs you didn't attach), we see that a couple of
the tests that exercise the same code path did in fact pass, while others
timed out.  I wouldn't consider that a regression.

On Mon, Nov 28, 2016 at 12:54 PM, Vinod Kone  wrote:

> +1 (binding)
>
> Tested on ASF CI.
>
>
> *Revision*: 52a0b0a41482da35dc736ec2fd445b6099e7a4e7
>
>- refs/tags/0.28.3-rc1
>
> Configuration Matrix gcc clang
> centos:7 --verbose --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> --verbose autotools
> [image: Success]
> 
> [image: Not run]
> cmake
> [image: Success]
> 
> [image: Not run]
> ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
> [image: Success]
> 
> [image: Success]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 
> --verbose autotools
> [image: Success]
> 
> [image: Success]
> 
> cmake
> [image: Success]
> 
> [image: Success]
> 
>
> On Mon, Nov 28, 2016 at 3:14 AM, Alex Rukletsov 
> wrote:
>
>> I see LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem failing on
>> CentOS 7 and Fedora 23, see e.g., [1]. I don't see any backports touching
>> [2], can it be a regression or this test is know to be problematic in
>> 0.28.x?
>>
>> [1] http://pastebin.com/c5PzfGF8
>> [2]
>> https://github.com/apache/mesos/blob/0.28.x/src/tests/contai
>> nerizer/filesystem_isolator_tests.cpp
>>
>> On Thu, Nov 24, 2016 at 

Slight change on Isolator interface on Linux

[Jie Yu]

Hi folks,

I just landed MESOS-6648 ,
which greatly simplified the launch path in MesosContainerizer.

The patch chain made MesosContainerizer launch helper to take
ContainerLaunchInfo. Prior to this patch, the launch helper takes various
flags from MesosContainerizer to launch the container. This makes it very
hard to add more parameters to the launch helper. This patch chain
simplifies this by passing 'ContainerLaunchInfo' instead.
'ContainerLaunchInfo' is also the protobuf message returned by isolators
during 'prepare()'. This makes it very easy to merge them and send it to
the launch helper. More importantly, this makes it very easy to add more
parameters to the launch helper in the future.

Because of this change, there is a slight change to ContainerLaunchInfo. We
changed the namespaces fields to `repeated`. This makes it easy for the
caller to merge ContainerLaunchInfo (simply leverage protobuf merge
functionality).

```
   // (Linux only) The namespaces a nested container
   // should enter in its parent before cloning.
-  optional uint32 enter_namespaces = 9 [default = 0];
+  repeated int32 enter_namespaces = 9;

   // (Linux only) The namespaces that should be cloned for the container.
   // The namespaces are created while launching the executor.
-  optional uint32 clone_namespaces = 4 [default = 0];
+  repeated int32 clone_namespaces = 4;
```

The isolator change to adapt to this is simple, simply a one line change
line the following:

```
-  launchInfo.set_clone_namespaces(CLONE_NEWPID);
+  launchInfo.add_clone_namespaces(CLONE_NEWPID);
```

- Jie

Re: Hook Deprecation: slavePreLaunchDockerEnvironmentDecorator

[Adam Bordelon]

+ dev@, in case some module developers haven't joined the modules@ list yet.
So this will require code changes (and a recompile) so any module
implementing the previous hook must now use the new hook, potentially
updating to set both environments instead of just the one. Such a hook
compiled against Mesos 1.1 will not work anymore with Mesos 1.2.

On Tue, Nov 29, 2016 at 10:15 AM, Till Toenshoff  wrote:

> Hey Folks,
>
> we are removing the slave sided, docker specific hook callback `
> slavePreLaunchDockerEnvironmentDecorator` and replace it with `
> slavePreLaunchDockerTaskExecutorDecorator`.
>
> The latter now allows setting two individual environments, one effective
> for the task and one for the executor. Note that as usual, a custom
> executor itself may still decide to make (parts of) the executor
> environment available to its task/s.
>
> We decided to use a protobuf message return signature for that hook,
> allowing us to add things like e.g. volumes without having to come up with
> another hook callback — see discussion around https://issues.apache.org/
> jira/browse/MESOS-6396?focusedCommentId=15654316=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15654316.
>
> The changes involved should have no extra side-effects - your
> functionality is unaffected if you don't use custom hooks.
>
> Please remember that hooks are still considered experimental and may still
> evolve fast.
>
>
> Let me know what you think!
>
> Till

[GitHub] mesos issue #190: Ensure curl is present on Ubuntu

[jieyu]

Github user jieyu commented on the issue:

https://github.com/apache/mesos/pull/190
  
Yeah, curl currently is a dependency if people wants to use container image 
support in unified containerizer. There is plan to remove this dependency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] mesos issue #190: Ensure curl is present on Ubuntu

[haosdent]

Github user haosdent commented on the issue:

https://github.com/apache/mesos/pull/190
  
The associate mail: [Making 'curl' a prerequisite for installing 
Mesos](https://www.mail-archive.com/dev@mesos.apache.org/msg34530.html)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] mesos issue #190: Ensure curl is present on Ubuntu

[vinodkone]

Github user vinodkone commented on the issue:

https://github.com/apache/mesos/pull/190
  
Thanks for the PR. 

@jieyu  what's our recommendation regarding curl dependency? I remember 
there was a discussion in the mailing list sometime back.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: MESOS-6233 Allow agents to re-register post a host reboot

[James Peach]

> On Nov 28, 2016, at 6:09 PM, Yan Xu  wrote:
> 
> So one thing that was brought up during offline conversations was that if the 
> host reboot is associated with hardware change (e.g., a new memory stick):
> 
>   • Currently: the agent would skip the recovery (and the chance of 
> running into incompatible agent info) and register as a new agent.
>   • With the change: the agent could run into incompatible agent info due 
> to resource change and flap indefinitely until the operator intervenes.
> 
> To mitigate this and maintain the current behavior, we can have the agent 
> remove `rm -f /meta/slaves/latest` automatically upon recovery 
> failure but only after the host has rebooted. This way the agent can restart 
> as a new agent without operator intervention. 
> 
> Any thoughts?

I still think you need a mechanism for the master/agent to tell you whether it 
will honor the restart policy. Without this, you have to lock the framework to 
a Mesos version.

An empty RestartPolicy is also problematic since it precludes using 
RestartPolicy in pods. If you later want to restart a task inside a pod but not 
across agent restarts you would have no way to express that.

J

Re: Design for Restartable Tasks

[Megha Sharma]

Hi All,

Thanks for your feedback on the design, here’s the revised design for 
Restartable Tasks.

https://docs.google.com/document/d/1epYCznSjevbiA776Yr72xx365IGEzKRnoLnXBorm0J0/edit?usp=sharing
 


Based on the feedback we have had on the old design and discussion with a bunch 
of committers, we have added the restart by executor and taskgroup restart to 
the design. Looking forward to your comments/feedback.

Many Thanks
Megha Sharma

> On Oct 26, 2016, at 6:23 PM, Benjamin Mahler  > wrote:
> 
> Thanks for publishing this! Saw some tickets being created and was wondering 
> where this email was.. :)
> 
> The higher level thing that strikes me is that I think the notion of a task 
> restart policy should be managed by the executor (i.e. the executor restarts 
> the task based on the policy). This is aligned with how the existing kill and 
> health check policies work. This project seems to be something more along the 
> lines of a restartable executor, alongside a change to perform agent recovery 
> across reboot?
> 
> Since this project is pretty complicated, it would be prudent to gather some 
> committers to provide feedback and we can publish our notes to the lists.
> 
> Ben
> 
> On Wed, Oct 26, 2016 at 5:13 PM, Megha Sharma  > wrote:
> Hi All,
> 
> We have been working on the design to allow tasks which need to be restarted 
> on the agent post its restart. Looking forward to your comments/feedback.
> 
> Design Doc:
> https://docs.google.com/document/d/1YS_EBUNLkzpSru0dwn_hPUIeTATiWckSaosXSIaHUCo/edit#heading=h.tlevdyt3yv0a
>  
> 
> 
> JIRA:
> https://issues.apache.org/jira/browse/MESOS-3545 
> 
> 
> Many Thanks
> Megha Sharma
> 
> 
> 
> 
> 

[GitHub] mesos pull request #190: Ensure curl is present on Ubuntu

[CVTJNII]

GitHub user CVTJNII opened a pull request:

https://github.com/apache/mesos/pull/190

Ensure curl is present on Ubuntu

When running on a clean Ubuntu install curl may not be present.  The 
following error was seen while experimenting with the ubuntu:14.04 Docker 
container:

```
Received status update TASK_FAILED for task 'test'
  message: 'Failed to launch container: Failed to perform 'curl': ABORT: 
(../../../3rdparty/libprocess/include/process/posix/subprocess.hpp:214): Failed 
to os::execvpe on path 'curl': No such file or directory
*** Aborted at 1480435360 (unix time) try "date -d @1480435360" if you are 
using GNU date ***
PC: @ 0x7f1d40711c37 (unknown)
*** SIGABRT (@0x61) received by PID 97 (TID 0x7f1d38ef5700) from PID 97; 
stack trace: ***
@ 0x7f1d40fca330 (unknown)
@ 0x7f1d40711c37 (unknown)
@ 0x7f1d40715028 (unknown)
@   0x46e1fc _Abort()
@   0x46e22a _Abort()
@ 0x7f1d4648d0e1 process::internal::childMain()
@ 0x7f1d46491f7b 
_ZNSt5_BindIFPFiRKSsPPcS3_RKN7process10Subprocess2IO20InputFileDescriptorsERKNS6_21OutputFileDescriptorsESC_bPiRKSt6vectorINS5_9ChildHookESaISF_EEESsS3_S3_S7_SA_SA_bSD_SH_EE6__callIiJEJLm0ELm1ELm2ELm3ELm4ELm5ELm6ELm7ELm8T_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x7f1d46491394 
_ZNSt5_BindIFPFiRKSsPPcS3_RKN7process10Subprocess2IO20InputFileDescriptorsERKNS6_21OutputFileDescriptorsESC_bPiRKSt6vectorINS5_9ChildHookESaISF_EEESsS3_S3_S7_SA_SA_bSD_SH_EEclIJEiEET0_DpOT_
@ 0x7f1d464904f0 std::_Function_handler<>::_M_invoke()
@   0x4780b0 std::function<>::operator()()
@ 0x7f1d4524f325 process::defaultClone()
@ 0x7f1d4648fb3d std::_Function_handler<>::_M_invoke()
@ 0x7f1d4648f3c3 std::function<>::operator()()
@ 0x7f1d4648d93d process::internal::cloneChild()
@ 0x7f1d4648b8c5 process::subprocess()
@ 0x7f1d45b967f1 mesos::uri::curl()
@ 0x7f1d45b96e18 mesos::uri::curl()
@ 0x7f1d45b991b8 mesos::uri::DockerFetcherPluginProcess::fetch()
@ 0x7f1d45ba485a 
_ZZN7process8dispatchI7NothingN5mesos3uri26DockerFetcherPluginProcessERKNS2_3URIERKSsS5_SsEENS_6FutureIT_EERKNS_3PIDIT0_EEMSE_FSC_T1_T2_ET3_T4_ENKUlPNS_11ProcessBaseEE_clESP_
@ 0x7f1d45baba29 
_ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchI7NothingN5mesos3uri26DockerFetcherPluginProcessERKNS6_3URIERKSsS9_SsEENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSI_FSG_T1_T2_ET3_T4_EUlS2_E_E9_M_invokeERKSt9_Any_dataS2_
@ 0x7f1d46441071 std::function<>::operator()()
@ 0x7f1d46423db1 process::ProcessBase::visit()
@ 0x7f1d4642c1c6 process::DispatchEvent::visit()
@ 0x7f1d4504084c process::ProcessBase::serve()
@ 0x7f1d464200a6 process::ProcessManager::resume()
@ 0x7f1d4641cd31 
_ZZN7process14ProcessManager12init_threadsEvENKUt_clEv
@ 0x7f1d4642b96e 
_ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEE9_M_invokeIIEEEvSt12_Index_tupleIIXspT_EEE
@ 0x7f1d4642b8c5 
_ZNSt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEclEv
@ 0x7f1d4642b85e 
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
@ 0x7f1d40d67a60 (unknown)
@ 0x7f1d40fc2184 start_thread
@ 0x7f1d407d537d (unknown)
' 
  source: SOURCE_AGENT
  reason: REASON_CONTAINER_LAUNCH_FAILED
```

This is resolved by running 'apt-get install curl' which this PR adds to 
the getting started instructions.

This PR only adds Curl to the Ubuntu instructions as it is believed to 
already be present on CentOS:

```shell
$ for dist in ubuntu:14.04 ubuntu:16.04 centos:6.6 centos:7; do echo 
"$dist: $(docker run $dist curl --version 2>&1 | head -n 1)"; done
ubuntu:14.04: docker: Error response from daemon: oci runtime error: exec: 
"curl": executable file not found in $PATH.
ubuntu:16.04: docker: Error response from daemon: oci runtime error: exec: 
"curl": executable file not found in $PATH.
centos:6.6: curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 
zlib/1.2.3 libidn/1.18 libssh2/1.4.2
centos:7: curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 
Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/CVTJNII/mesos Docs_Install_Curl_Ubuntu

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mesos/pull/190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #190


commit 5d207098b7d1eeff143e08ef2207417683408f4f
Author: Tom Noonan II