Re: HAProxy for Hive Thrift Server on Mesos

[xiaokun]

After set the timeout value, the tcp request become stable.

2015-03-23 23:26 GMT+08:00 John Omernik :

> So per my HA-Proxy logs, I am getting a condition where the client
> side time expired while waiting for the client to send or receive
> data, and this is occurring during the DATA phase. (the two letter
> HAProxy code is cD . To test, I expanded the timeout (on all things)
> to 10.
>
> This was interesting in that it showed me the problem a bit clearer.
> Now, as long as I submit and results for a query in the 100 second
> range, my connection stays open and happy.  If I let it sit for 100
> seconds, and then try a query, then it breaks again.
>
> It appears HA proxy gets irritable when things aren't being sent over
> a connection for a long period of time.  For services like Minecraft,
> this is not an issue, when a client is connected, there is always data
> there, and no issues occur. For Thrift, there may be multiple long
> pauses as an analyst thinks through the next query etc.  I wonder what
> would be the best way to handle that on a per container level. (not
> all containers need this sort of customization...)
>
>
>
> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun  wrote:
> > I have a similar error when launch jenkins in mesos cluster. And I use
> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
> time
> > to time, connection between jenkins master and jenkins slave will lost,
> > while will cause jobs fail.
> >
> > I will try to set timeout value for HAProxy
> >
> > 2015-03-23 4:18 GMT+08:00 John Omernik :
> >>
> >> Yep all three are going through HAProxy and seem to be working well.
> >> I should do some testing to determine if I run a thrift server that
> >> connects to metastore in HAProxy to determine if that's the broken
> >> link, but I believe it's between the thrift server. I will check out
> >> the link you posted to get a better idea out of how it works. I am
> >> running this all on a MapR cluster which is Great for Docker/Mesos as
> >> I used the MapR FS, mounted locally on each node's NFS to put the data
> >> for MySQL, the logs for the metastore/thift server, and my world files
> >> for Minecraft.  So far it's all really slick.
> >>
> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
> >>  wrote:
> >> > It sounds like you may be running into the connection timeout that is
> >> > configured for the HAProxy instance.
> >> >
> >> > You can see an instance of this in Marathon's
> >> > HAProxy-marathon-bridge[1].
> >> > You may need to raise the timeout values.
> >> >
> >> > Are you accessing MySql, Hive metastore and minecraft through HAProxy
> as
> >> > well? I know that Minecraft running in docker on mesos works well,
> >> > though
> >> > I've always used a direct connection not a proxied one.
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
> >> >
> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik 
> wrote:
> >> >>
> >> >> I have a nice setup with a Hive thrift server running in a docker
> >> >> container on Mesos. It works pretty well, but something, I believe in
> >> >> how HAProxy works with the connection, is causing the thrift server
> >> >> connection  to die after a time.  Basically, I can run a few queries,
> >> >> but after 2 or 3, or specifically after a longer query, I get the
> >> >> error below indicating End of File on the connection.  Then no more
> >> >> connections work until I reestablish the connection to the thrift
> >> >> server. I've tried looking in logs. the thrift server std err logs
> >> >> show no issues. I am guess I need to dig into haproxy logs, but I am
> >> >> not seeing any issues in syslog so far.  I'd love any pointers on how
> >> >> to trouble shoot this.  By the way, I have MySQl, hive metastore, and
> >> >> a minecraft server all running on Mesos/Docker with no issues, not
> >> >> sure why the thrift server is so sensitive.
> >> >>
> >> >> :)
> >> >>
> >> >>
> >> >> pyhs2 connection string:
> >> >>
> >> >> hs2 =
> >> >>
> >> >>
> pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
> >> >>
> >> >>
> >> >> Error
> >> >>
> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
> in
> >> >> read(self, sz)
> >> >> 118 if len(buff) == 0:
> >> >> 119   raise
> >> >> TTransportException(type=TTransportException.END_OF_FILE,
> >> >> --> 120 message='TSocket read 0
> bytes')
> >> >> 121 return buff
> >> >> 122
> >> >>
> >> >> TTransportException: TSocket read 0 bytes
> >> >
> >> >
> >
> >
>

Re: [RESULT][VOTE] Release Apache Mesos 0.22.0 (rc4)

[craig w]

Congrats, I was working on a quick post summarizing what's new (based on
jira and the video from niklas) which I just posted (great timing)

http://craigwickesser.com/2015/03/mesos-022-release/

On Tue, Mar 24, 2015 at 8:30 PM, Paul Otto  wrote:

> This is awesome! Thanks for all the hard work you all have put into this!
> I am really excited to update to the latest stable version of Apache Mesos!
>
> Regards,
> Paul
>
>
> Paul Otto
> Principal DevOps Architect, Co-founder
> Otto Ops LLC | *OttoOps.com *
> 970.343.4561 office
> 720.381.2383 cell
>
> On Tue, Mar 24, 2015 at 6:04 PM, Niklas Nielsen 
> wrote:
>
>> Hi all,
>>
>> The vote for Mesos 0.22.0 (rc4) has passed with the
>> following votes.
>>
>> +1 (Binding)
>> --
>> Ben Mahler
>> Tim St Clair
>> Adam Bordelon
>> Brenden Matthews
>>
>> +1 (Non-binding)
>> --
>> Alex Rukletsov
>> Craig W
>> Ben Whitehead
>> Elizabeth Lingg
>> Dario Rexin
>> Jeff Schroeder
>> Michael Park
>> Alexander Rojas
>> Andrew Langhorn
>>
>> There were no 0 or -1 votes.
>>
>> Please find the release at:
>> https://dist.apache.org/repos/dist/release/mesos/0.22.0
>>
>> It is recommended to use a mirror to download the release:
>> http://www.apache.org/dyn/closer.cgi
>>
>> The CHANGELOG for the release is available at:
>>
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0
>>
>> The mesos-0.22.0.jar has been released to:
>> https://repository.apache.org
>>
>> The website (http://mesos.apache.org) will be updated shortly to reflect
>> this release.
>>
>> Thanks,
>> Niklas
>>
>
>


-- 

https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links

Re: [RESULT][VOTE] Release Apache Mesos 0.22.0 (rc4)

[Paul Otto]

This is awesome! Thanks for all the hard work you all have put into this! I
am really excited to update to the latest stable version of Apache Mesos!

Regards,
Paul


Paul Otto
Principal DevOps Architect, Co-founder
Otto Ops LLC | *OttoOps.com *
970.343.4561 office
720.381.2383 cell

On Tue, Mar 24, 2015 at 6:04 PM, Niklas Nielsen 
wrote:

> Hi all,
>
> The vote for Mesos 0.22.0 (rc4) has passed with the
> following votes.
>
> +1 (Binding)
> --
> Ben Mahler
> Tim St Clair
> Adam Bordelon
> Brenden Matthews
>
> +1 (Non-binding)
> --
> Alex Rukletsov
> Craig W
> Ben Whitehead
> Elizabeth Lingg
> Dario Rexin
> Jeff Schroeder
> Michael Park
> Alexander Rojas
> Andrew Langhorn
>
> There were no 0 or -1 votes.
>
> Please find the release at:
> https://dist.apache.org/repos/dist/release/mesos/0.22.0
>
> It is recommended to use a mirror to download the release:
> http://www.apache.org/dyn/closer.cgi
>
> The CHANGELOG for the release is available at:
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0
>
> The mesos-0.22.0.jar has been released to:
> https://repository.apache.org
>
> The website (http://mesos.apache.org) will be updated shortly to reflect
> this release.
>
> Thanks,
> Niklas
>

[RESULT][VOTE] Release Apache Mesos 0.22.0 (rc4)

[Niklas Nielsen]

Hi all,

The vote for Mesos 0.22.0 (rc4) has passed with the
following votes.

+1 (Binding)
--
Ben Mahler
Tim St Clair
Adam Bordelon
Brenden Matthews

+1 (Non-binding)
--
Alex Rukletsov
Craig W
Ben Whitehead
Elizabeth Lingg
Dario Rexin
Jeff Schroeder
Michael Park
Alexander Rojas
Andrew Langhorn

There were no 0 or -1 votes.

Please find the release at:
https://dist.apache.org/repos/dist/release/mesos/0.22.0

It is recommended to use a mirror to download the release:
http://www.apache.org/dyn/closer.cgi

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0

The mesos-0.22.0.jar has been released to:
https://repository.apache.org

The website (http://mesos.apache.org) will be updated shortly to reflect
this release.

Thanks,
Niklas

Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)

[Tim St Clair]

+1 binding Fedora 21 GCC 4.9.2 

- Original Message -

> From: "Andrew Langhorn" 
> To: user@mesos.apache.org
> Cc: cc...@tampabay.rr.com
> Sent: Tuesday, March 24, 2015 7:05:37 AM
> Subject: RE: [VOTE] Release Apache Mesos 0.22.0 (rc4)

> +1 (non-binding) on Ubuntu 14.04 with GCC 4.8.2​

> From: Alexander Rojas 
> Sent: 24 March 2015 11:02
> To: user@mesos.apache.org
> Cc: cc...@tampabay.rr.com
> Subject: Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
> +1 (non binding)

> > On 24 Mar 2015, at 07:27, Michael Park < mcyp...@gmail.com > wrote:
> 

> > +1 (non-binding)
> 

> > make check passes on:
> 

> > * Mac OS X 10.10.2 + clang 3.5.0
> 
> > * Ubuntu 14.04 + gcc 4.4.7
> 
> > * Ubuntu 14.04 + gcc 4.6.4
> 
> > * Ubuntu 14.04 + gcc 4.8.2
> 

> > On 24 March 2015 at 01:01, Niklas Nielsen < nik...@mesosphere.io > wrote:
> 

> > > I would like to steer this conversation/thread back to the release vote.
> > > Can
> > > you restart the distcc discussion in another thread?
> > 
> 

> > > Cheers,
> > 
> 
> > > Niklas
> > 
> 

> > > On 23 March 2015 at 21:31, CCAAT < cc...@tampabay.rr.com > wrote:
> > 
> 

> > > > Excellent start! Nice links I was not aware of (thanks).
> > > 
> > 
> 
> > > > Folks can use distcc now to test new rollouts of mesos?
> > > 
> > 
> 
> > > > That was the quest of the thread was/is to establish some codes for
> > > 
> > 
> 
> > > > testing new rollouts of mesos.
> > > 
> > 
> 

> > > > It'll need to be 'extended' for cross compiling too for my needs.
> > > 
> > 
> 
> > > > I'd like to follow up with anyone that get's distcc working with cross
> > > > compiling for different arches. arm64 would be really cool. (LLVM?)
> > > 
> > 
> 

> > > > My small cluster needs some work, much of it not related to mesos, but
> > > 
> > 
> 
> > > > my efforts to run mesos and spark without the HDFS and use Cephfs,
> > > > btrfs
> > > > and
> > > > supporting codes. So I'm not sure when I'll get this mesos-distcc
> > > > installed
> > > > and running in the near future; but I am most interested in follow the
> > > > issues other encounter on compiling and cross compiling on a mesos
> > > > cluster.
> > > 
> > 
> 

> > > > What would be really cool is to run this distcc-mesos on top of spark
> > > 
> > 
> 
> > > > and cephfs for some real fast compile times of large codes.
> > > 
> > 
> 

> > > > Thanks,
> > > 
> > 
> 
> > > > James
> > > 
> > 
> 

> > > > On 03/23/2015 10:22 PM, Adam Bordelon wrote:
> > > 
> > 
> 

> > > > > I know it's over a year old and hasn't been updated, but bmahler
> > > > > already
> > > > 
> > > 
> > 
> 
> > > > > created a distcc framework example for Mesos.
> > > > 
> > > 
> > 
> 
> > > > > https://github.com/mesos/ mesos-distcc
> > > > 
> > > 
> > 
> 

> > > > > On Mon, Mar 23, 2015 at 7:56 PM, CCAAT < cc...@tampabay.rr.com
> > > > 
> > > 
> > 
> 
> > > > >  > wrote:
> > > > 
> > > 
> > 
> 

> > > > > On 03/23/2015 09:02 PM, Adam Bordelon wrote:
> > > > 
> > > 
> > 
> 

> > > > > Integration tests are definitely desired/recommended. Some of us
> > > > 
> > > 
> > 
> 
> > > > > devs
> > > > 
> > > 
> > 
> 
> > > > > just do make [dist]check, but others test integrations with their
> > > > 
> > > 
> > 
> 
> > > > > favourite frameworks, or push it to their internal testing clusters.
> > > > 
> > > 
> > 
> 
> > > > > We're open to any additional testing you would like to propose
> > > > 
> > > 
> > 
> 
> > > > > and/or
> > > > 
> > > 
> > 
> 
> > > > > perform.
> > > > 
> > > 
> > 
> 

> > > > > Thanks,
> > > > 
> > > 
> > 
> 
> > > > > -Adam-
> > > > 
> > > 
> > 
> 

> > > > > Distcc is a distributed compiling program that has been a long staple
> > > > > on
> > > > 
> > > 
> > 
> 
> > > > > Gentoo linux and many other distros. I work on Gentoo and I think
> > > > > that
> > > > 
> > > 
> > 
> 
> > > > > setting up distcc to run on a mesos cluster is a fabulous idea. Not
> > > > > only
> > > > 
> > > 
> > 
> 
> > > > > can you compile native binaries for the inherent arch, but cross
> > > > 
> > > 
> > 
> 
> > > > > compiling should work too. Everyone has to recompile (optimized)
> > > > > kernels
> > > > 
> > > 
> > 
> 
> > > > > frequently with the release cycle of the linux kernel team. With the
> > > > 
> > > 
> > 
> 
> > > > > current roll out of all sorts of 64 bit arm systems, there's going to
> > > > > be
> > > > 
> > > 
> > 
> 
> > > > > a great opportunity to cross compile arm64 bit codes on CISC (X86_64)
> > > > 
> > > 
> > 
> 
> > > > > bit clusters.
> > > > 
> > > 
> > 
> 

> > > > > This also starts the process of heterogeneous mesos clusters, surely
> > > > 
> > > 
> > 
> 
> > > > > inevitable.
> > > > 
> > > 
> > 
> 

> > > > > https://code.google.com/p/__ distcc/ < https://code.google.com/p/
> > > > > distcc/
> > > > > >
> > > > 
> > > 
> > 
> 

> > > > > https://code.google.com/p/__ distcc/downloads/list
> > > > 
> > > 
> > 
> 
> > > > > < https://code.google.com/p/ distcc/downloads/list >
> > > > 
> > > 
> > 
> 

> > > > > With LLvm, gnu and other proje

Re: Mesos slaves connecting but not active.

[Giulio Eulisse]

Ciao,

I updated to 0.21.1 and seems to have fixed the issue (at least the 
slave reconnects). docker is still slow deleting stuff.


--
Ciao,
Giulio

On 23 Mar 2015, at 18:20, Tim Chen wrote:


How many containers are you running, and what is your system like?

Also are you able to capture through perf or strace what docker rm is
blocked on?

Tim


On Mon, Mar 23, 2015 at 10:12 AM, Giulio Eulisse 


wrote:

I suspect my problem is that "docker rm" takes forever in my case. 
I'm not

running docker in docker though.


On 23 Mar 2015, at 18:01, haosdent wrote:

Are your issue relevant to this?

https://issues.apache.org/jira/browse/MESOS-2115

On Tue, Mar 24, 2015 at 12:52 AM, Giulio Eulisse 


wrote:

Hi,


I'm running using 0.20.1 and I seem to have troubles due to the 
fact a
mesos slave is not able to recover the docker containers after a 
restart,

resulting in a very long wait.

Is this some known issue?

--
Ciao,
Giulio





--
Best Regards,
Haosdent Huang

RE: [VOTE] Release Apache Mesos 0.22.0 (rc4)

[Andrew Langhorn]

+1 (non-binding) on Ubuntu 14.04 with GCC 4.8.2?



From: Alexander Rojas 
Sent: 24 March 2015 11:02
To: user@mesos.apache.org
Cc: cc...@tampabay.rr.com
Subject: Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)

+1 (non binding)

On 24 Mar 2015, at 07:27, Michael Park 
mailto:mcyp...@gmail.com>> wrote:

+1 (non-binding)

make check passes on:

  *   Mac OS X 10.10.2 + clang 3.5.0
  *   Ubuntu 14.04 + gcc 4.4.7
  *   Ubuntu 14.04 + gcc 4.6.4
  *   Ubuntu 14.04 + gcc 4.8.2

On 24 March 2015 at 01:01, Niklas Nielsen 
mailto:nik...@mesosphere.io>> wrote:
I would like to steer this conversation/thread back to the release vote. Can 
you restart the distcc discussion in another thread?

Cheers,
Niklas

On 23 March 2015 at 21:31, CCAAT 
mailto:cc...@tampabay.rr.com>> wrote:
Excellent start! Nice links I was not aware of (thanks).
 Folks can use distcc now to test new rollouts of mesos?
That was the quest of the thread was/is to establish some codes for
testing new rollouts of mesos.

It'll need to be 'extended' for cross compiling too for my needs.
I'd like to follow up with anyone that get's distcc working with cross 
compiling for different arches. arm64 would be really cool. (LLVM?)

My small cluster needs some work, much of it not related to mesos, but
my efforts to run mesos and spark without the HDFS and use Cephfs, btrfs and 
supporting codes. So I'm not sure when I'll get this mesos-distcc installed and 
running in the near future; but I am most interested in follow the issues other 
encounter on compiling and cross compiling on a mesos cluster.

What would be really cool is to run this distcc-mesos on top of spark
and cephfs for some real fast compile times of large codes.


Thanks,
James


On 03/23/2015 10:22 PM, Adam Bordelon wrote:
I know it's over a year old and hasn't been updated, but bmahler already
created a distcc framework example for Mesos.
https://github.com/mesos/mesos-distcc

On Mon, Mar 23, 2015 at 7:56 PM, CCAAT 
mailto:cc...@tampabay.rr.com>
>> wrote:

On 03/23/2015 09:02 PM, Adam Bordelon wrote:

Integration tests are definitely desired/recommended. Some of us
devs
just do make [dist]check, but others test integrations with their
favourite frameworks, or push it to their internal testing clusters.
We're open to any additional testing you would like to propose
and/or
perform.

Thanks,
-Adam-


Distcc is a distributed compiling program that has been a long staple on
Gentoo linux and many other distros. I work on Gentoo and I think that
setting up distcc to run on a mesos cluster is a fabulous idea. Not only
can you compile native binaries for the inherent arch, but cross
compiling should work too. Everyone has to recompile (optimized) kernels
frequently with the release cycle of the linux kernel team. With the
current roll out of all sorts of 64 bit arm systems, there's going to be
a great opportunity to cross compile arm64 bit codes on CISC (X86_64)
bit clusters.

This also starts the process of heterogeneous mesos clusters, surely
inevitable.

https://code.google.com/p/__distcc/ 

https://code.google.com/p/__distcc/downloads/list


With LLvm, gnu and other projects, compiling and cross compiling on
a mesos cluster is sure to be a very, very popoular idea. Any CI
endeavor
will necessitate lots of compiling too.

hope this helps,
James

Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)

[Alexander Rojas]

+1 (non binding)

> On 24 Mar 2015, at 07:27, Michael Park  wrote:
> 
> +1 (non-binding)
> 
> make check passes on:
> Mac OS X 10.10.2 + clang 3.5.0
> Ubuntu 14.04 + gcc 4.4.7
> Ubuntu 14.04 + gcc 4.6.4
> Ubuntu 14.04 + gcc 4.8.2
> 
> On 24 March 2015 at 01:01, Niklas Nielsen  > wrote:
> I would like to steer this conversation/thread back to the release vote. Can 
> you restart the distcc discussion in another thread?
> 
> Cheers,
> Niklas
> 
> On 23 March 2015 at 21:31, CCAAT  > wrote:
> Excellent start! Nice links I was not aware of (thanks).
>  Folks can use distcc now to test new rollouts of mesos?
> That was the quest of the thread was/is to establish some codes for
> testing new rollouts of mesos.
> 
> It'll need to be 'extended' for cross compiling too for my needs.
> I'd like to follow up with anyone that get's distcc working with cross 
> compiling for different arches. arm64 would be really cool. (LLVM?)
> 
> My small cluster needs some work, much of it not related to mesos, but
> my efforts to run mesos and spark without the HDFS and use Cephfs, btrfs and 
> supporting codes. So I'm not sure when I'll get this mesos-distcc installed 
> and running in the near future; but I am most interested in follow the issues 
> other encounter on compiling and cross compiling on a mesos cluster.
> 
> What would be really cool is to run this distcc-mesos on top of spark
> and cephfs for some real fast compile times of large codes.
> 
> 
> Thanks,
> James
> 
> 
> On 03/23/2015 10:22 PM, Adam Bordelon wrote:
> I know it's over a year old and hasn't been updated, but bmahler already
> created a distcc framework example for Mesos.
> https://github.com/mesos/mesos-distcc 
> 
> On Mon, Mar 23, 2015 at 7:56 PM, CCAAT  
> >> wrote:
> 
> On 03/23/2015 09:02 PM, Adam Bordelon wrote:
> 
> Integration tests are definitely desired/recommended. Some of us
> devs
> just do make [dist]check, but others test integrations with their
> favourite frameworks, or push it to their internal testing clusters.
> We're open to any additional testing you would like to propose
> and/or
> perform.
> 
> Thanks,
> -Adam-
> 
> 
> Distcc is a distributed compiling program that has been a long staple on
> Gentoo linux and many other distros. I work on Gentoo and I think that
> setting up distcc to run on a mesos cluster is a fabulous idea. Not only
> can you compile native binaries for the inherent arch, but cross
> compiling should work too. Everyone has to recompile (optimized) kernels
> frequently with the release cycle of the linux kernel team. With the
> current roll out of all sorts of 64 bit arm systems, there's going to be
> a great opportunity to cross compile arm64 bit codes on CISC (X86_64)
> bit clusters.
> 
> This also starts the process of heterogeneous mesos clusters, surely
> inevitable.
> 
> https://code.google.com/p/__distcc/  
> >
> 
> https://code.google.com/p/__distcc/downloads/list 
> 
>  >
> 
> With LLvm, gnu and other projects, compiling and cross compiling on
> a mesos cluster is sure to be a very, very popoular idea. Any CI
> endeavor
> will necessitate lots of compiling too.
> 
> hope this helps,
> James
> 
> 
> 
> 
> 
> 
>