Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

483/#comment101354> This change makes fetcher in line with slave/main.cpp or master/main.cpp and initialized the logging. - Ankur Chauhan On Nov. 5, 2014, 7:57 p.m., Ankur Chauhan wrote: > > --- > This is an automatically gener

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

d7754009a59fedb43e3422c56b3a786ce80164aa Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Logging warning in mesos fetcher

Will do! -- Ankur > On 5 Nov 2014, at 10:56, Vinod Kone wrote: > > Good catch. You want to add the log initialization to the main() in > fetcher.cpp. See master/main.cpp or slave/main.cpp for examples. > > On Wed, Nov 5, 2014 at 10:32 AM, Ankur Chauhan wrote: > >>

Logging warning in mesos fetcher

I noticed in a run of bin/mesos-tests.sh --gtest_filter=FetcherTest.*: $ ./bin/mesos-tests.sh --gtest_filter=FetcherTest.* Source directory: /Users/achauhan/Projects/mesos Build directory: /Users/achauhan/Projects/mesos/build - We cannot

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

omains/example";>More information... ``` - Ankur --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27483/#review59938 --------

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

ore InitGoogleLogging() is written to STDERR ``` And looking at the code, I don't see a glog initialization codeblock. Should there be `google::InitGoogleLogging(argv[0]);` in the `main` method? - Ankur Chauhan On Nov. 5, 2014, 6:39 a.m., Ankur Cha

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

> On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote: > > src/launcher/fetcher.cpp, line 223 > > <https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223> > > > > Why fall through if result.isError()? > > Ankur Chauhan wrote: > The

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

> On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote: > > src/launcher/fetcher.cpp, line 223 > > <https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223> > > > > Why fall through if result.isError()? > > Ankur Chauhan wrote: > The

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27483/#review59842 ------- On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote: > >

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

> On Nov. 3, 2014, 9:59 p.m., Timothy St. Clair wrote: > > src/launcher/fetcher.cpp, line 77 > > <https://reviews.apache.org/r/27483/diff/1/?file=746870#file746870line77> > > > > we should probably bail here, if somehow the return is != 0 (isError() >

Re: Why rely on url scheme for fetching?

plicated > enough to not cause me major problems no matter when it lands. > > Bernd > >> On Nov 4, 2014, at 11:44 AM, Ankur Chauhan wrote: >> >> Hi ben, >> >> Thanks for the follow up. I think thats a perfectly fine game plan. I think >> I c

Re: Why rely on url scheme for fetching?

>>> extensible, and more testable. Anyone that's heard me rant knows I am a big >>> fan of unit tests over integration tests, so this shouldn't surprise anyone >>> :) >>> >>> If you haven't already, please read the documentation on contri

Re: Why rely on url scheme for fetching?

>>> extensible, and more testable. Anyone that's heard me rant knows I am a big >>> fan of unit tests over integration tests, so this shouldn't surprise anyone >>> :) >>> >>> If you haven't already, please read the documentation on contri

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

) - src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

fs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

return fetchWithHadoopClient(); > > > > ``` I am pretty sure that native cp is faster than hadoop client local copy. I can make this change. - Ankur --- This is an automatically generated e-mail

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

) - src/hdfs/hdfs.hpp bbfeddef106c598d8379ced085ef0605c4b2f380 src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

------- On Nov. 3, 2014, 1:52 a.m., Ankur Chauhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27483/ > --

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

-- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27483/#review59653 ----------- On Nov. 3, 2014, 1:52 a.m., Ankur Chauhan wrote: > > -

Re: Why rely on url scheme for fetching?

ded, since it's going to take some > rounds of review, and Ian is cutting the release today. > > We'll have to tag this for the next release. > > Tim > >> On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan wrote: >> Hi Tim/others, >> >> Is this to

Re: Why rely on url scheme for fetching?

ded, since it's going to take some > rounds of review, and Ian is cutting the release today. > > We'll have to tag this for the next release. > > Tim > >> On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan wrote: >> Hi Tim/others, >> >> Is this to

Re: Why rely on url scheme for fetching?

(s) as Target Version = 0.22.0. > > Ian > > On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > >> Hi Tim/others, >> >> Is this to be included in the 0.21.0 release? If so, I don't know how to >> tag it etc.

Re: Why rely on url scheme for fetching?

(s) as Target Version = 0.22.0. > > Ian > > On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > >> Hi Tim/others, >> >> Is this to be included in the 0.21.0 release? If so, I don't know how to >> tag it etc.

Re: Why rely on url scheme for fetching?

n contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to reviewboard to get it reviewed and committed. >> >> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan > <mailto:an...@malloc64.com>> wrote: >

Re: Why rely on url scheme for fetching?

n contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to reviewboard to get it reviewed and committed. >> >> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan > <mailto:an...@malloc64.com>> wrote: >

Re: Why rely on url scheme for fetching?

nit tests over integration tests, so this shouldn't surprise anyone >> :) >> >> If you haven't already, please read the documentation on contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to

Re: Why rely on url scheme for fetching?

nit tests over integration tests, so this shouldn't surprise anyone >> :) >> >> If you haven't already, please read the documentation on contributing to >> Mesos and the style guide to ensure all the naming is as expected, then you >> can push the patch to

Re: Why rely on url scheme for fetching?

mments and/or suggestions. -- Ankur @ankurcha > On 2 Nov 2014, at 18:17, Ankur Chauhan wrote: > > Hi, > > I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is > pretty coarse grained and are more on the lines of a functional test. I was > going to

Re: AWS - Master and Slave can't communicate

Custom TCP Rule - TCP - 2888 - 0.0.0.0/0 > Custom TCP Rule - TCP - 3888 - 0.0.0.0/0 > > > I don't know if this is correct or I've missed to open some other port. > >> On Mon, Nov 3, 2014 at 2:55 PM, Ankur Chauhan wrote: >> Are you running VPC? If so, what are

Re: AWS - Master and Slave can't communicate

Are you running VPC? If so, what are your routing /security group rules ? -- Ankur > On 2 Nov 2014, at 19:27, Matt Ivankov wrote: > > Hi, > > I'm new to Mesos and Marathon and I'd like to install it on AWS. For start, I > wanted to create 1 master and 1 slave (Mesos version 0.20.1). > > I've

Re: Why rely on url scheme for fetching?

it very soon. > I know Tim St. Clair had some ideas for fixing MESOS-1711 > <https://issues.apache.org/jira/browse/MESOS-1711>; he may want to review too. > > On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > Hi Tim, > > I jus

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

bbfeddef106c598d8379ced085ef0605c4b2f380 src/launcher/fetcher.cpp 9323c28237010fa065ef34d74435c151ded530a8 Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/mesos-style.py Thanks, Ankur Chauhan

Re: Why rely on url scheme for fetching?

. -- Ankur > On 1 Nov 2014, at 20:05, Ankur Chauhan wrote: > > Yea, i saw that the minute i pressed send. I'll start the review board so > that people can have a look at the change. > > -- Ankur > >> On 1 Nov 2014, at 20:01, Tim Chen > <mailto:t...@mesosphe

Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

(updated) --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Why rely on url scheme for fetching?

Yea, i saw that the minute i pressed send. I'll start the review board so that people can have a look at the change. -- Ankur > On 1 Nov 2014, at 20:01, Tim Chen wrote: > > Hi Ankur, > > There is a fetcher_tests.cpp in src/tests. > > Tim > > On Sat, Nov 1

Re: Why rely on url scheme for fetching?

Hi Ankur, > > Can you post on reviewboard? We can discuss more about the code there. > > Tim > > Sent from my iPhone > > On Nov 1, 2014, at 6:29 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > >> Hi Tim, >> >> I don't think ther

Re: Why rely on url scheme for fetching?

31 October 2014 22:39, Tim Chen <mailto:t...@mesosphere.io>> wrote: > I believe there is already a JIRA ticket for this, if you search for fetcher > in Mesos JIRA I think you can find it. > > Tim > > On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan <mailto:an...@mall

Re: build failure on mac os x

r us not > having done that, yet. > > Let us know if that solved your problem. > > Till > >> On Nov 1, 2014, at 10:03 AM, Ankur Chauhan > <mailto:an...@malloc64.com>> wrote: >> >> I am trying to build the latest (master) mesos source and keep getting

build failure on mac os x

I am trying to build the latest (master) mesos source and keep getting this error: In file included from ../../src/state/log.cpp:25:0: ../../3rdparty/libprocess/3rdparty/stout/include/stout/svn.hpp:21:23: fatal error: svn_delta.h: No such file or directory #include ^ com

Why rely on url scheme for fetching?

Hi, I have been looking at some of the stuff around the fetcher and saw something interesting. The code for fetcher::fetch method is dependent on a hard coded list of url schemes. No doubt that this works but is very restrictive. Hadoop/HDFS in general is pretty flexible when it comes to being

Re: CDH5.2.3 on mesos

the property > hadoop.skip.worker.version.check to True - In the later case I wish you good > luck... Neverhteless, I am not sure if this error can also appear in other > cases. > > Hope this helps. > > Stratos > > >> On Tue, Oct 28, 2014 at 12:30 PM, Ankur

Re: CDH5.2.3 on mesos

Anyone else have something to add on this? -- Ankur Chauhan > On 28 Oct 2014, at 02:10, Ankur Chauhan wrote: > > Hi tom, > > I was basically following the readme. This gist has the list of commands how > i am setting up things > https://gist.github.com/ankurcha/a9504b0e4

Re: CDH5.2.3 on mesos

hare your hadoop configuration and perhaps a > description of the problems you're having? > > Tom. > > > > On Tue, Oct 28, 2014 at 8:53 AM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > > H, > > > I was trying to setup mesos/hadoop with t

CDH5.2.3 on mesos

H, I was trying to setup mesos/hadoop with the latest CDH version (MR1) and it seems like the instructions are sort of out of date and I also tried the suggestions in https://github.com/mesos/hadoop/issues/25 but after 4 hours of flailing around I am

Re: Docker: pull on app start?

I had a similar issue and what I ended up doing was to explicitly set the version tag instead of just saying :latest. That makes the whole system much more stable/predictable. For the cost of a single curl call when you push is much better for debugging issues and the state of the system. -- Ank

Re: Exposing host services in docker container

> Try the following instead: > HOST_IP=`hostname -i` > docker run -it --rm dockerfile/mongodb bash -c "mongo --host $HOST_IP:27017" > > Tim > > On Sun, Oct 26, 2014 at 11:28 PM, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > Hi Tim, > > Thanks

Re: Exposing host services in docker container

container should be able to access any > port in the host, if it's cross hosts then you have to setup your own bridge > and use the lxc-conf option, or use something like pipework. > > I'm adding the lxc-conf options into the next release of mesos. > > Tim > >

Exposing host services in docker container

Hi all, So I have been dabbling with docker containers in and mesos. I have the following scenario and i was wondering if someone had experience with something like this. I have a bunch of dockerized apps running and the slave hosts have some common services running, something like mongos and

Re: Do i really need HDFS?

<< "' to '" << path << "'"; >Try result = hdfs.copyToLocal(uri, path); > if (result.isError()) { > LOG(ERROR) << "HDFS copyToLocal failed: " << result.error(); > return Error(result.error()); >} > --

Re: Do i really need HDFS?

This is what I also intend to do. Is a s3 path considered non-hdfs? If so, how does it know the credentials to use to fetch the file. Sent from my iPhone > On Oct 21, 2014, at 5:16 AM, David Greenberg wrote: > > We use spark without HDFS--in our case, we just use ansible to copy the spark >

Re: Do i really need HDFS?

t;> these required 'feature sets' is of interest to many. >> >> >> James >> >> >> >> >>> On 10/20/14 07:08, David Greenberg wrote: >>> You certainly don't need hdfs if you've got other infrastructure th

Do i really need HDFS?

P_HOME and get away with it? It would really help how other have their mesos setup in production or what they would recommend regarding my setup? --Ankur Chauhan

Re: Review Request 26736: Ensure docker calls stop if a docker_stop_timeout is provided Added the flag docker_stop_timeout that defaults to 0, if the timeout is 0 the a docker kill will be run, otherw

> On Oct. 17, 2014, 11:02 p.m., Mesos ReviewBot wrote: > > Bad patch! > > > > Reviews applied: [26736] > > > > Failed command: ./support/apply-review.sh -n -r 26736 > > > > Error: > > --2014-10-17 23:02:08-- https://reviews.apache.org/r/26736/diff/raw/ > > Resolving reviews.apache.org (revie

[jira] [Commented] (MESOS-1925) Docker kill does not allow containers to exit gracefully

[ https://issues.apache.org/jira/browse/MESOS-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175680#comment-14175680 ] Ankur Chauhan commented on MESOS-1925: -- Patch here: https://reviews.apache.o

Re: Review Request 26736: Ensure docker calls stop if a docker_stop_timeout is provided Added the flag docker_stop_timeout that defaults to 0, if the timeout is 0 the a docker kill will be run, otherw

> On Oct. 15, 2014, 10:47 p.m., Mesos ReviewBot wrote: > > Bad patch! > > > > Reviews applied: [26736] > > > > Failed command: git apply --index 26736.patch > > > > Error: > > error: patch failed: src/docker/docker.cpp:472 > > error: src/docker/docker.cpp: patch does not apply > > error: patc

Re: Review Request 26736: Ensure docker calls stop if a docker_stop_timeout is provided Added the flag docker_stop_timeout that defaults to 0, if the timeout is 0 the a docker kill will be run, otherw

IGTERM gracefully degrades to a kill it fails. We could simplify the code further by always calling stop and completely dropping the kill code path. - Ankur Chauhan On Oct. 15, 2014, 10:58 a.m., Ryan Thomas wrote: > > ---

Re: Killing Docker containers

is something we want to make configurable. > > ryan > > On 15 October 2014 08:32, Ankur Chauhan <mailto:an...@malloc64.com>> wrote: > ++ I was planning on submitting that patch. But if someone has this sorted > out already, I'll defer. > > Sent from my

Re: Review Request 26709: Change from kill to stop for docker

> On Oct. 15, 2014, 12:15 a.m., Timothy Chen wrote: > > src/docker/docker.cpp, line 472 > > > > > > I'm not sure what executor timeout you're referring to, but for example > > the executor signal escalation timeout al

Re: Review Request 26709: Change from kill to stop for docker

> On Oct. 14, 2014, 11:39 p.m., Ankur Chauhan wrote: > > src/docker/docker.cpp, line 472 > > <https://reviews.apache.org/r/26709/diff/1/?file=721122#file721122line472> > > > > A 30 second timeout seems pretty arbitrary. I would suggest allowing >

Re: Review Request 26709: Change from kill to stop for docker

> On Oct. 14, 2014, 11:39 p.m., Ankur Chauhan wrote: > > src/docker/docker.cpp, line 472 > > <https://reviews.apache.org/r/26709/diff/1/?file=721122#file721122line472> > > > > A 30 second timeout seems pretty arbitrary. I would suggest allowing >

Re: Review Request 26709: Change from kill to stop for docker

ng the environment variable. 3. If nothing is set, kill as usual. The only reason why i suggest these is beacuse a lot of people may already have the expectation that killing a task will kill the container. This way, the upgrade process is a smooth upgrade flow. - Ankur Chauhan

Re: Killing Docker containers

++ I was planning on submitting that patch. But if someone has this sorted out already, I'll defer. Sent from my iPhone > On Oct 14, 2014, at 2:19 PM, Ryan Thomas wrote: > > The docker stop command will attempt to kill the container if it doesn't stop > in 10 seconds by default. I think we s

Re: Multiple Network interfaces

Is using bridged networking with docker an option. That would map the service port (fixed) to a dynamically assigned port. Granted it will not utilize different nics but it's something. Sent from my iPhone > On Oct 8, 2014, at 8:52 AM, Jay Buffington wrote: > > My reading of the code is that

Re: Potential Seattle Mesos User Group

++ u would like to help. Contact the me directly and we can work out things. Sent from my iPhone > On Sep 21, 2014, at 10:29 AM, Dave Lester wrote: > > Hi All, > > I've chatted with a few different people (including some folks in the Twitter > Seattle office) about potentially starting a Sea

Re: Potential Seattle Mesos User Group

++ I would like to help. Sent from my iPhone > On Sep 21, 2014, at 10:29 AM, Dave Lester wrote: > > Hi All, > > I've chatted with a few different people (including some folks in the Twitter > Seattle office) about potentially starting a Seattle Mesos User Group. Does > anyone else want to h

Re: Frontend loadbalancer configuration for long running tasks

x27;d be > interested to see if this solution works for you! > > Thanks, > Tom > > On Thu, Sep 11, 2014 at 3:37 AM, Ankur Chauhan wrote: > >> I looked around a bit but the docs on aurora are almost non-existant when >> you go into anything more that the basics. Sing

Re: Frontend loadbalancer configuration for long running tasks

ody, > > Ankur's post is general enough, allowing me to reiterate the question: > > Does anyone know about similar HAProxy solutions for Aurora? > > Thanks, > Stephan > > > On Di, 2014-09-09 at 01:52 -0700, Ankur Chauhan wrote: > > Hi all, > > > >

Re: Frontend loadbalancer configuration for long running tasks

. As always, any comments suggestions are welcome! -- Ankur Chauhan On Tue, Sep 9, 2014 at 11:54 AM, Mike Michel wrote: > Hi, > > > > i changed the haproxy script so it makes use of haproxys ACL feature. I > need to have subdomains for every new app a user creates but always th

Re: Frontend loadbalancer configuration for long running tasks

se can be found at > https://github.com/opencredo/mesos_service_discovery > > The main advantage of this project is that it doesn't depend on DNS, so it > avoids issues with expiring TTLs and application caches, which can be > problematic. > > Cheers, > Bart &g

Re: Frontend loadbalancer configuration for long running tasks

/Opensourcing-Bamboo > > Cheers, > > Ryan > On 9 Sep 2014 18:53, "Ankur Chauhan" wrote: > >> Hi all, >> >> (Please let me know if this is not the correct place for such a question). >> I have been looking at mesos + marathon + haproxy as a way of deployin

Frontend loadbalancer configuration for long running tasks

Hi all, (Please let me know if this is not the correct place for such a question). I have been looking at mesos + marathon + haproxy as a way of deploying long running web applications. Mesos coupled with marathon's /tasks api gives me all the information needed to get a haproxy configured and loa

Logging for containers

in it. Something like /mnt/logs//taskId_.log Option 2: send the logs to syslog/rsyslog and use a centralized logging server of sorts Are there any other options? -- Ankur Chauhan

Re: Introducing Portainer

Could you share a link to the repo? On Wed, Sep 3, 2014 at 3:20 PM, Tom Arnfeld wrote: > Hey everyone, > > Thought it would be worth sharing this on the mailing list. We've recently > open sourced a Mesos framework called Portainer, which is for *building* > docker containers on top of your clu

Re: MongoDB on mesos

much as hoping a whole replica set is not moved. > Aurora, for example, implements this with a 'dedicated' scheduling > constraint specified in the job description. > > -=Bill > > > On Tue, Sep 2, 2014 at 9:01 PM, Ankur Chauhan > wrote: > >> Hi Vinod, &g

Re: MongoDB on mesos

work in this > area. > > > On Tue, Sep 2, 2014 at 2:50 PM, Ankur Chauhan wrote: > >> Hi all, >> >> I apologise for the repost but wanted to get some responses. I am evaluating >> mesos and wanted to know if anyone has successfully used >> mesos (with or

MongoDB on mesos

Hi all, I apologise for the repost but wanted to get some responses. I am evaluating mesos and wanted to know if anyone has successfully used mesos (with or without docker) as a means of managing a mongodb deployment. Currently, I have 8 shards where each shard is a 3x replicated replicaset and co

MongoDB on mesos

Hi all, I am evaluating mesos and wanted to know if anyone has successfully used mesos (with or without docker) as a means of managing a mongodb deployment. Currently, I have 8 shards where each shard is a 3x replicated replicaset and contains > 500GB of data. I was wondering if someone has tried

Re: Mesos autoscaling script

Looks cool! I'll check it out. Sent from my iPhone > On Aug 27, 2014, at 2:00 PM, Michael Babineau > wrote: > > I wrote a script to automatically launch/terminate Mesos slaves on EC2 based > on available resources: > https://github.com/thefactory/autoscale-python/blob/master/examples/mesos_e

Re: unsubscribe

I, for one, really liked google groups. Nabble is pretty painful to use. > On Feb 8, 2014, at 16:27, Mayur Rustagi wrote: > > Recently moved the mailing list from Google and deactivated that one :) > Just send a mail to user-unsubscr...@spark.incubator.apache.org > > > > Mayur Rustagi > Ph

Re: Python API Performance

How does Julia interact with spark. I would be interested, mainly because I seem to find scala syntax a little obscure and it would be great to see actual numbers comparing scala, Python, Julia workloads. > On Feb 1, 2014, at 16:08, Aureliano Buendia wrote: > > A much (much) better solution t

Re: Hadoop MapReduce on Spark

I think the whole idea of the spark API is to simplify building iterative workflows/algorithms when compared to Hadoop's bloated API I am not saying it's completely wrong or anything although it would be clearer if you had a particular use case in mind that you wish to tackle. > On Feb 1, 201

Re: Quality of documentation (rant)

Hi ognen, I am in the same boat as you are. I actually work on a project that basically does this exact same thing and tried to aggregations by spark but it ended up with me trying again and again to do simple things like reading from s3 and such and failing. I have a little bit of knowledge a

Re: Which of the hadoop file formats are supported by Spark ?

You may also want to consider Parquet (http://parquet.io). It is pretty efficient http://zenfractal.com/2013/08/21/a-powerful-big-data-trio/ -- Ankur Chauhan

Re: Mesos(phere) chef cookbook

This is great! This was probably 80% of the reason I couldn't push for mesos in my cluster! > On Jan 16, 2014, at 10:17, Vinod Kone wrote: > > Looks good! > > > @vinodkone > > >> On Thu, Jan 16, 2014 at 6:53 AM, Connor Doyle wrote: >> This is great news! >> Thanks Ray, >> -- >> Connor Doy

Re: Porting an app

at our deployment strategies, so I may not have been super clear about some stuff or over simplified some critical points. Please let me know what you think about this, I would love to contribute to marathon/mesos although I am pretty green as far as scala is concerned. Java is my strong su

Re: Porting an app

Thanks everyone for all the help. Marathon does seem like a good framework but my use case requires the app to evaluate it's own health and scale up based on internal load stats (SLA requirements) and I don't know if marathon supports that. This is the main reason why i am looking at building out m

Re: Please help: virtualization type 'hvm' when I try to launch ec2 ssd instance

Hi, I ran into the same problem, the AMI that is used but he EC2 scripts is a paravirtualization one. For i2 and c3 nodes you need the newer HVM virtualization. I think the ec2-scripts don't support HVM instances. I ended up manually setting spark or you could try using the chef recipe to do

Porting an app

Hi all, I just got started on mesos and I am trying to port a netty based webapp to use a mesos as a framework. I wrote a Executor and a Scheduler. But I am not quite sure as to how to go about getting everything to work together. I can't really find an example that is an end-to-end use case. B

Re: Bump: on disk storage formats

t; > On Sun, Dec 8, 2013 at 3:03 PM, Andrew Ash wrote: >> LZO compression at a minimum, and using Parquet as a second step, >> seems like the way to go though I haven't tried either personally yet. >> >> Sent from my mobile phone >> >> On Dec 8, 2013, a

Bump: on disk storage formats

Hi all, Sorry for posting this again but I am interested in finding out what different on disk data formats for storing timeline event and analytics aggregate data. Currently I am just using newline delimited json gzipped files. I was wondering if there were any recommendations. -- Ankur

data storage formats

Hi all, I am wondering what do people use as the on disk storage format. I have seen almost all the examples use csv files to store and load data but that seems too simplisting for obvious reasons (compressibility to name one). I was just interested to find out what people use to store computat

Re: Opinions stratosphere

s for > joins (broadcast/repartition, sort-merge/hybrid-hash join). Another > difference is that its operators are designed to work in memory but > gracefully go out of core under memory pressure. > > Checkout the feature overview on the start page of http://stratosphere.eu/ > >

Re: How to run compile .jar in spark?

Have a look at https://github.com/apache/incubator-spark/blob/master/run-example That should help you figure out how to run a jar file using spark (given you have the classpath/dependencies set up). -- Ankur On 25 Nov 2013, at 14:12, Michael Kun Yang wrote: > How to run compile .jar in spark?

<    1   2   3   >