Cos Indeed It's more critical to File Systems and Databases for having SPOF compared to computing engines. >> >> might crash in 5 months or in 2 minutes. If the failure is caused by a hardware issue, then it's unlikely to have No.2 going down in just 2 minutes. Although most of the time in our case they're software issues, which is why there's a case that No.2 goes down immediately after No.1 dies. Due to that, I see the HA feature in the way that it at least solve part of the question. Probably those gossip, no master implementations are more robust, I don't have any experience though.
Faraaz Thanks for the answer and looking for the announcement :) 2016-01-27 11:04 GMT+08:00 RJ Nowling <[email protected]>: > Oops... meant QFS, not Apache Apex. Sorry! But the question about having > a maintainer still stands. :) > > On Tue, Jan 26, 2016 at 9:02 PM, RJ Nowling <[email protected]> wrote: > > > Hi Faraaz, > > > > Is there someone who would be willing to take responsibility for > > maintaining the Apache Apex packages in Bigtop? > > > > We generally like to find a maintainer for each package and add them to > > our maintainers list. This way we know who to contact if a build fails > and > > blocks a release. If there is no maintainer (and the build has > problems), > > we use that as grounds for removing the package from Bigtop. > > > > Thanks! > > RJ > > > > On Tue, Jan 26, 2016 at 3:05 PM, Faraaz Sareshwala < > > [email protected]> wrote: > > > >> Thanks for the warm welcome guys :). > >> > >> I agree that smaller patches will be easier to parse and review. Right > >> now, I have Debian and RPM packaging complete. I have created a JIRA > ticket > >> (https://issues.apache.org/jira/browse/BIGTOP-2283) which I will use to > >> track the integration. > >> > >> I’ll start sending in pull requests as I complete each of the tasks > >> necessary to get this integration done. I’d love all of your guys’ > guidance > >> along the way so that the integration can be as correct to bigtop’s > >> standards as possible. > >> > >> Also, regarding the single point of failure sidenote discussed in > another > >> thread, Quantcast is actively working on updating qfs to support a > >> distributed metaserver so that we can avoid the single point of failure. > >> It’s not ready for release yet but we will announce it when it is :). > >> > >> > >> Faraaz > >> > >> On 1/26/16, 11:33 AM, "jay vyas" <[email protected]> wrote: > >> > >> >Please add Your name to the MAINTAINERS.txt file in the PR for "QFS" ! > >> > > >> >On Tue, Jan 26, 2016 at 2:24 PM, Konstantin Boudnik <[email protected]> > >> wrote: > >> > > >> >> On Wed, Jan 27, 2016 at 01:05AM, Evans Ye wrote: > >> >> > Oh, if I made you uncomfortable, you know I didn't mean that :) > >> >> > >> >> Oh believe me - it takes a lot more than that to make me > uncomfortable > >> ;) > >> >> Besides, you gave me an opportunity to speak up about the SPOFs in > the > >> >> Hadoop > >> >> ecosystem. Which I can do for hours! > >> >> > >> >> > I just want to know more about the design because I'm always > >> interested > >> >> in > >> >> > the architecture things, especially when I see something different. > >> >> > Even Spark just added HA in their master in recent release last > >> year. So > >> >> > it's totally fine. > >> >> > > >> >> > And the following is just for the discussion: > >> >> > > >> >> >> Just a side node: as you know literally _all_ components in > >> Hadoop-based > >> >> >> stack have SPOF. > >> >> > > >> >> > I think it's more related to the maturity of a component, but > >> somehow it > >> >> > will get there to eliminate the SPOF just like what HDFS, YARN, > >> Oozie, > >> >> Spark > >> >> > did. For operations, two SPOFs is good enough for them to have a > >> good > >> >> sleep > >> >> > at night ;) > >> >> > >> >> It might be ok for Oozie, but not for a file system. Because once > your > >> >> master > >> >> has failed over to the standby you don't know how long that one has. > It > >> >> might > >> >> crash in 5 months or in 2 minutes. That's why if you're an admin for > a > >> >> production critical cluster you'll get paged immediately. Perhaps 3 > >> >> o'clock in > >> >> the morning. > >> >> > >> >> I spent last three years working on the active-active software for > >> mission > >> >> critical systems, so I would love to share some architectural > thoughts > >> with > >> >> you, if you're interested. > >> >> > >> >> Cos > >> >> > >> >> > 2016-01-26 12:08 GMT+08:00 Jay Vyas <[email protected]>: > >> >> > > >> >> > > Awesome! It's great to hear about another HDFS alternative :) > >> >> > > > >> >> > > , We tried pretty hard to make bigtop more "HCFS" compliant, by > >> adding > >> >> the > >> >> > > init hcfs tooling and so on, but never completed making the > puppet > >> >> recipes > >> >> > > add support for other file systems. > >> >> > > > >> >> > > > >> >> > > Would be very interesting to see if the quant cast team can take > >> over > >> >> on > >> >> > > the HCFS integration so that bigtop is a home for more Than just > >> one > >> >> under > >> >> > > filesystem . > >> >> > > > >> >> > > Let us know what your planning on working on, looking forward to > >> it...! > >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > On Jan 25, 2016, at 9:06 PM, Faraaz Sareshwala < > >> >> > > [email protected]> wrote: > >> >> > > > > >> >> > > > Hello everyone in the Bigtop community! > >> >> > > > > >> >> > > > My name is Faraaz Sareshwala and I am an engineer working at > >> >> Quantcast > >> >> > > to integrate QFS (https://quantcast.github.io/qfs) into the > Bigtop > >> >> > > project. I just wanted to send a small note out to introduce > >> myself to > >> >> the > >> >> > > community and announce that we are working on an integration. > Feel > >> >> free to > >> >> > > reach out to me if you have any questions about QFS, our > >> integration, > >> >> or > >> >> > > Quantcast itself. > >> >> > > > > >> >> > > > At this point, I am nearly complete integrating QFS. I plan to > >> >> submit a > >> >> > > patch through github and JIRA later in the week or early next > week > >> >> once I > >> >> > > get a chance to test everything is working correctly. I’ve read > the > >> >> wiki on > >> >> > > contribution to the project, but if you guys have any other tips > or > >> >> > > guidance for me, I’d really appreciate that. > >> >> > > > > >> >> > > > Patch coming soon! > >> >> > > > > >> >> > > > Take care, > >> >> > > > Faraaz > >> >> > > > > >> >> > > > >> >> > >> > > >> > > >> > > >> >-- > >> >jay vyas > >> > > > > >
