Just submitted (20 mins + questions): Apache Bigtop at the Wikimedia Foundation
Apache Bigtop is an Apache Software Foundation project for packaging, deploying, and testing open source big data components. The Analytics/Data-Engineering team at the Wikimedia Foundation has been collaborating with the project to move from its Cloudera CDH 5.x Hadoop distribution to Apache Bigtop. This talk highlights the work done with upstream to support the Wikimedia's use case (for example, adding support for Debian 10) and the migration/upgrade plan devised. Moving away from CDH is not easy, especially in the Wikimedia ecosystem where several tools (mostly Apache based) gravitate around Hadoop (and hence compatibility needs to be preserved): Druid, Spark, Kafka, Camus/Gobblin, Oozie, Hive, Presto, Jupyter Notebooks, etc..). Luca On Tue, Dec 15, 2020 at 4:08 PM Yuqi Gu <[email protected]> wrote: > > Thanks, Kengo and Evans. > How about to include topics as follows: > > - Implemented a new Ambari Management Packs (Mpack) that enables Ambari > to remove the tight coupling with existing stacks (HDP, HDF, etc.) from > Ambari core and flexibly manage any Hadoop-related service into Ambari by > leveraging Bigtop as the Software packages stack, to make Hadoop deployment > and management simpler. > - Introduced the ELK into Bigtop Stack. > > Thanks. > > BRs, > Yuqi > > > On Tue, 15 Dec 2020 at 13:07, Kengo Seki <[email protected]> wrote: > > > Thank you for the notification, Evans! > > I've just submitted the following proposal. Luca and Yuqi, if you > > don't make it, let's include your talk into mine and update the title > > and abstract. > > > > -------------------------------- > > > > Title: What's new in Bigtop 1.5 > > > > Short Abstract: > > Apache Bigtop is an Apache Software Foundation project for packaging, > > deploying, and testing open source big data components. Its latest > > version 1.5.0 has been released in Dec 2020. > > In this talk, the speakers provides an overview of the recent upgrade > > and some basic usages from users' point of view, for example: > > > > * Installing big data components such as Hadoop/Spark/Kafka as deb or > > rpm packages from Bigtop's public repository > > * Automate deploying components using Puppet manifests provided by Bigtop > > * Verify the deployed cluster using smoke tests provided by Bigtop > > * Create a local cluster on VMs or containers using Bigtop provisioners > > > > Kengo Seki <[email protected]> > > > > On Tue, Dec 15, 2020 at 2:50 AM Evans Ye <[email protected]> wrote: > > > > > > The deadline seems super tight (TODAY![1]). I suggest folks who're > > > interested in doing the talk just go for a submission and then we can > > > discuss whether to have a joint talk back in the mailing list. > > > > > > 1. Bigtop at Wikimedia > > > 2. Bigtop MPack > > > 3. What's new in Bigtop 1.5 > > > > > > If we can't make it. We can still shoot for ApacheCon 2021 (though > > there's > > > no timeline yet). > > > > > > Anyhow, glad to see folks are interested in doing talks on Bigtop! > > > > > > [1] https://lists.fosdem.org/pipermail/fosdem/2020q4/003112.html > > > > > > > > > Kengo Seki <[email protected]> 於 2020年12月14日 週一 下午2:59寫道： > > > > > > > Sounds great! Masatake and I are also interested in the joint talk. > > > > We'd like to introduce Bigtop itself breifly, 1.5.0 upgrade overview > > > > and its basic usages, > > > > if there's no other folks who'd like to talk the same theme. > > > > > > > > Kengo Seki <[email protected]> > > > > > > > > On Mon, Dec 14, 2020 at 11:29 AM Yuqi Gu <[email protected]> wrote: > > > > > > > > > > *>>> Folks, >>> Any one interested in the joint talk? Though I'm not > > > > > directly involved in >>> the development. I see 1.5 is a high quality > > > > > release and worth sharing the >>> story. We can also add a section > > > > > outlining the future of Bigtop.* > > > > > > > > > > That's great. I'd like to share the Bigtop-Mpack feature in our new > > 1.5 > > > > > release and be glad to participate in this joint talk. > > > > > > > > > > BRs, > > > > > Yuqi > > > > > > > > > > > > > > > > > > > > On Mon, 14 Dec 2020 at 02:34, Evans Ye <[email protected]> wrote: > > > > > > > > > > > That's really a great idea! I think a joint talk of Bigtop 1.5 > > release > > > > and > > > > > > the real-world case from Wikimedia would be a powerful talk to > > deliver. > > > > > > > > > > > > Luca, > > > > > > Are you comfortable to share the case on FOSDEM21? I remember > > that we > > > > had > > > > > > a similar discussion before and I think this is a good stage to > > > > present. > > > > > > > > > > > > Folks, > > > > > > Any one interested in the joint talk? Though I'm not directly > > involved > > > > in > > > > > > the development. I see 1.5 is a high quality release and worth > > sharing > > > > the > > > > > > story. We can also add a section outlining the future of Bigtop. > > > > > > > > > > > > Evans > > > > > > > > > > > > > > > > > > > > > > > > Konstantin Boudnik <[email protected]> 於 2020年12月14日 週一 上午12:18寫道： > > > > > > > > > > > > > That'd be actually real great! I am in a middle of the move > > > > (again!), so > > > > > > > I can't promise anything of a sort. But having something from a > > > > > > > real-world application would be super to submit and present! > > > > > > > -- > > > > > > > With regards, > > > > > > > Cos > > > > > > > > > > > > > > On 13.12.2020 05:53, Olaf Flebbe wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > Anyone to promote Bigtop at FOSDEM’21 (virtual) ? > > > > > > > > > > > > > > > > Would be really great if we could get a Bigtop Version 1.5 > > Upgrade > > > > Talk > > > > > > > or even a Bigtop@Wikimedia (Luca?) presented. > > > > > > > > FOSDEM is the biggest OSS event in Europe, by large. > > > > > > > > > > > > > > > > Best > > > > > > > > Olaf > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Anfang der weitergeleiteten Nachricht: > > > > > > > >> > > > > > > > >> Von: Kenneth Hoste <[email protected]> > > > > > > > >> Betreff: HPC, Big Data and Data Science devroom at FOSDEM'21 > > > > > > > >> Datum: 12. Dezember 2020 um 11:43:25 MEZ > > > > > > > >> An: undisclosed-recipients: ; > > > > > > > >> > > > > > > > >> Dear open source enthusiast, > > > > > > > >> > > > > > > > >> This is a one time message to promote the HPC, Big Data, and > > Data > > > > > > > Science devroom at FOSDEM'21 (https://fosdem.org < > > > > https://fosdem.org/>). > > > > > > > >> > > > > > > > >> You are receiving this message because you have submitted a > > talk > > > > > > > proposal for one of the previous editions of the devroom, or have > > > > been > > > > > > > involved with the organization of it. > > > > > > > >> > > > > > > > >> FOSDEM'21 will be fully virtual, for well known reasons. > > > > > > > >> This makes the organization of the event and individual > > devrooms > > > > > > > particularly challenging. > > > > > > > >> > > > > > > > >> We hope that you are willing to help promote the HPC, Big > > Data, > > > > and > > > > > > > Data Science devroom at FOSDEM'21, or perhaps submit a talk > > proposal > > > > > > > yourself... > > > > > > > >> > > > > > > > >> The submission deadline is really close (Tue Dec 15th 2020, > > which > > > > may > > > > > > > get extended by a couple of days), but very light weight: it > > > > basically > > > > > > > comes down to a talk title + short description (no paper, etc.). > > > > > > > >> > > > > > > > >> The actual event is over the weekend of Feb 6-7 2021; > > > > > > > >> the HPC devroom itself is planned for Sunday Feb 7th 2021. > > > > > > > >> > > > > > > > >> Accepted talks will need to be pre-recorded by mid Jan'21. > > > > > > > >> Talks will be live streamed during the event with live Q&A & > > > > > > > discussions after each talk. > > > > > > > >> > > > > > > > >> Please visit https://hpc-bigdata-fosdem21.github.io < > > > > > > > https://hpc-bigdata-fosdem21.github.io/> for more information. > > > > > > > >> > > > > > > > >> Feel free to contact me in case of questions! > > > > > > > >> > > > > > > > >> Please consider forwarding this message to people or projects > > who > > > > may > > > > > > > be interested in submitting a talk proposal, either to the HPC > > > > devroom, > > > > > > or > > > > > > > one of the many other devrooms at FOSDEM'21 ( > > > > > > > https://fosdem.org/2021/schedule/tracks < > > > > > > > https://fosdem.org/2021/schedule/tracks>). > > > > > > > >> > > > > > > > >> > > > > > > > >> regards, > > > > > > > >> > > > > > > > >> Kenneth > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
