Re: Non-checkpointing frameworks

+1 to A and B Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 On 17 October 2016 at 00:38, Qian Zhang wrote: > and requires operators to enable checkpointing on the slaves. > > > Just curious why operator needs to enable checkpointing on the slave

Problems with shared libraries with mesos containerization....

Hi all, I've been working with the mesos containerizer to handle my docker containers recently. I created a docker container that requires libtiff.so.5, and the application within it runs well. But when I try to run it within the mesos containerizer I get an error saying libtiff.so.5 doesn't ex

Re: Problems with shared libraries with mesos containerization....

Is the library part of the image that you are running? Also you might need to setup LD_LIBRARY_PATH in your env while launching the image so that the container process knows where to look for the shared object. On Mon, Oct 17, 2016 at 5:21 AM, Mark Hammons wrote: > Hi all, > > I've been working

Re: Problems with shared libraries with mesos containerization....

Yes, it's installed under /usr/lib/x86_64 or whatever the multilib path is in debian. It seems files under this path are not accessible. I added LD_PRELOAD=/usr/lib/libtiff.so.5 to try to force the symlink to load but it refused to load it. I think the mesos containerizer is preventing the prog

Re: Problems with shared libraries with mesos containerization....

You are running a container with its own image right? So is /usr/lib/x86_64 in the container's file system or the host file system? On Mon, Oct 17, 2016 at 6:47 AM, Mark Hammons wrote: > Yes, it's installed under /usr/lib/x86_64 or whatever the multilib path is > in > debian. It seems files und

Re: Problems with shared libraries with mesos containerization....

Inside the container's file system. I was hoping that as far as my application is concerned my host file system doesn't exist. On Monday, October 17, 2016 6:52:17 AM CEST Avinash Sridharan wrote: > You are running a container with its own image right? So is /usr/lib/x86_64 > in the container's f

Re: Problems with shared libraries with mesos containerization....

Here's the code I define my executor to mesos with: val iuwtURI = CommandInfo.URI.newBuilder().setValue("http://***/ IUWT.tar.gz").setExtract(true).setCache(false).build() val iuwtjURI = CommandInfo.URI.newBuilder().setValue("http://***/ iuwtExecutor-assembly-0.1- SNAPSHOT.jar").s

Re: Problems with shared libraries with mesos containerization....

No, but even when I set LD_PRELOAD=/usr/lib/libtiff.so.5 for the process environment it says it can't load /usr/lib/libtiff.so.5. On Monday, October 17, 2016 7:52:16 AM CEST Avinash Sridharan wrote: > Are you setting the env in the dockerfile? > > On Mon, Oct 17, 2016 at 6:58 AM, Mark Hammons >

Re: Problems with shared libraries with mesos containerization....

Are you setting the env in the dockerfile? On Mon, Oct 17, 2016 at 6:58 AM, Mark Hammons wrote: > Here's the code I define my executor to mesos with: > > val iuwtURI = CommandInfo.URI.newBuilder().setValue("http://***/ > IUWT.tar.gz").setExtract(true).setCache(false).build() > >

Re: Problems with shared libraries with mesos containerization....

Can you prepend your command with `ls -al /usr/lib` and check in stdout if you are seeing the shared object? By the way why are you using LD_PRELOAD instead of LD_LIBRARY_PATH. LD_PRELOAD will force the linker to load your library before any other libraries. It's usually used to override system lib

Re: Problems with shared libraries with mesos containerization....

As I've shown in the original logs, the symbolic link libtiff.so.5 is in /usr/ lib. I used LD_PRELOAD just to try to force /usr/lib/libtiff.so.5 to be loaded since it wasn't for some reason. On Monday, October 17, 2016 8:03:34 AM CEST Avinash Sridharan wrote: > Can you prepend your command with `

Re: Problems with shared libraries with mesos containerization....

Adding LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu doesn't fix the error. On Monday, October 17, 2016 5:06:34 PM CEST Mark Hammons wrote: > As I've shown in the original logs, the symbolic link libtiff.so.5 is in > /usr/ lib. I used LD_PRELOAD just to try to force /usr/lib/libtiff.so.5 to > be loade

Re: Problems with shared libraries with mesos containerization....

No, it's a regular linux log. I've reattached it. On Monday, October 17, 2016 8:32:07 AM CEST Avinash Sridharan wrote: > can't seem to open the attached logs, is it gzip? > > On Mon, Oct 17, 2016 at 8:14 AM, Mark Hammons > wrote: > > > > Adding LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu doesn't

Re: Problems with shared libraries with mesos containerization....

can't seem to open the attached logs, is it gzip? On Mon, Oct 17, 2016 at 8:14 AM, Mark Hammons wrote: > Adding LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu doesn't fix the error. > > On Monday, October 17, 2016 5:06:34 PM CEST Mark Hammons wrote: > > As I've shown in the original logs, the symboli

Re: Problems with shared libraries with mesos containerization....

Does look like the symlink exists: /usr/lib/libtiff.so.5 I am assuming you have checked the realpath exists as well (that's why was asking for `ls -al`) ? Don't see that you have volume mounts that might obfuscate the path. Could you create a JIRA and if possible point give access to your docker

Re: Problems with shared libraries with mesos containerization....

Also would be good to see the env variables (LD_LIBRARY_PATH) being seen by the container when you start it, just to make sure the env are getting set correctly. On Mon, Oct 17, 2016 at 8:42 AM, Avinash Sridharan wrote: > Does look like the symlink exists: > /usr/lib/libtiff.so.5 > > I am assumi

Re: Problems with shared libraries with mesos containerization....

I've created the issue and attached a log file with all the info you asked for to it. https://issues.apache.org/jira/browse/MESOS-6404 On Monday, October 17, 2016 8:42:13 AM CEST Avinash Sridharan wrote: > Does look like the symlink exists: > /usr/lib/libtiff.so.5 > > I am assuming you have che

Re: Problems with shared libraries with mesos containerization....

Oh, the container image itself is very large. Do you have some place I can upload it to? On Monday, October 17, 2016 8:43:10 AM CEST Avinash Sridharan wrote: > Also would be good to see the env variables (LD_LIBRARY_PATH) being seen by > the container when you start it, just to make sure the env

Re: Problems with shared libraries with mesos containerization....

Maybe just attach the docker file and make sure layers are available in docker hub? On Mon, Oct 17, 2016 at 9:07 AM Mark Hammons wrote: > Oh, the container image itself is very large. Do you have some place I can > upload it to? > > On Monday, October 17, 2016 8:43:10 AM CEST Avinash Sridharan wr

Re: Non-checkpointing frameworks

+1 to both A to B. Do we plan to eventually drop non-checkpionted framework support (possibly in v2) and declare that all frameworks has to operate in this assumption? On Mon, Oct 17, 2016 at 1:36 AM, Aaron Carey wrote: > +1 to A and B > > Aaron Carey > Production Engineer - Cloud Pipeline > In

Re: Non-checkpointing frameworks

Qian, Turns out the --checkpoint flag was made default and removed in Mesos 0.22. On Sun, Oct 16, 2016 at 4:38 PM, Qian Zhang wrote: > and requires operators to enable checkpointing on the slaves. > > > Just curious why operator needs to enable checkpointing on the slaves (I > do not see an age

Re: Performance regression in v1 api vs v0

Dario, It's not immediately clear to me where the bottleneck might be. I filed MESOS-6405 to write a benchmark that tries to mimic your test setup and then go about fixing the issues. -anand On Sun, Oct 16, 2016 at 6:20 PM, Dario Rexin wrote:

Re: Performance regression in v1 api vs v0

Hi Anand, thanks for creating the ticket. I will also investigate a bit more. I will probably be in SF on Thursday, so we could discuss in person. -- Dario > On Oct 17, 2016, at 12:19 PM, Anand Mazumdar wrote: > > Dario, > > It's not immediately clear to me where the bottleneck might be. I f

Re: Performance regression in v1 api vs v0

+1 We are also quite interested in this topic. On Mon, Oct 17, 2016 at 12:34 PM, Dario Rexin wrote: > Hi Anand, > > thanks for creating the ticket. I will also investigate a bit more. I will > probably be in SF on Thursday, so we could discuss in person. > > -- > Dario > > On Oct 17, 2016, at 1

Re: Non-checkpointing frameworks

Got it, thanks Zameer! Thanks, Qian Zhang On Tue, Oct 18, 2016 at 2:25 AM, Zameer Manji wrote: > Qian, > > Turns out the --checkpoint flag was made default and removed in Mesos 0.22. > > On Sun, Oct 16, 2016 at 4:38 PM, Qian Zhang wrote: > >> and requires operators to enable checkpointing on