Ambari is not easy to maintain for its technology stack. But it will be a good choice for bigtopers to deploy and monitor their components.I think we need a *web app (Ambari-like)* to operate hadoop and improve bigtop's user experience. *Maybe we can refactor Ambari to work well with bigtop components. *
Kengo Seki <[email protected]> 于2022年4月1日周五 15:20写道： > Hi Michiel, > > I really appreciate your offer, that sounds very useful for the HDP users. > But to be honest, I'm afraid I'm a bit reluctant to include it into Bigtop, > due to the following reasons. > > 1. Ambari has already been retired this January [1]. > Keeping retired software as a Bigtop component > brings us security risks and maintenance cost. > Especially, Ambari depends on several obsolete > softwares such as Python2 and Bower. > So I personally think we should drop Ambari > (and also Mpack) at some point in the future, > unless it revives. > > 2. Cloudera has changed the license of their product [2], > so I'm not sure if we could do that. Doesn't it violate > their license to install and use HDP without subscription? > If the new Mpack is derived from their product, > could we include it in our distributions > from the viewpoint of license compatibility [3]? > > But the above is just my personal opinion, so I'd like to hear from others. > > [1]: https://lists.apache.org/thread/m5jrpn4j28kn3wfn4zzxvy0g450vdlr1 > [2]: https://www.cloudera.com/downloads/paywall-expansion.html > [3]: https://www.apache.org/legal/resolved.html > > Kengo Seki <[email protected]> > > > On Sat, Mar 26, 2022 at 4:40 PM Michiel Verheul <[email protected]> wrote: > > > > Hi Kengo, > > > > I'm currently working on porting the Hortonworks HDP 3.1 mpack to make it > > compatible with Bigtop 3.0.0. > > Do you think it's an idea to include that as a separate (BGTP-3.0) mpack > in > > the bigtop repo? I've just reached the point that it installs from > Ambari, > > completes the setup wizard and runs a > > HDFS+YARN+Mapreduce2+Hive+Zookeeper+Spark cluster without any errors. > But a > > lot more testing has to be done to see if all services run nicely. > > Apart from that, I also want to enable the rest of the HDP Services that > > are also included in Bigtop. (HBase, Kafka, Flink, Zeppelin, etc) > > Please let me know what you think. > > > > > > Op za 26 mrt. 2022 07:38 schreef Kengo Seki <[email protected]>: > > > > > Hi Peng, > > > > > > Thank you so much for testing! > > > > > > > I find that the version of Bigtop3.0.0's bigtop-ambari-mpack is *not* > > > compatible with this version of Ambari (2.7.5), also is *not* > compatible > > > with the previous one (2.6.1, provided by Bigtop 1.5.0). > > > > > > I also confirmed that combination doesn't work. The Bigtop stack > > > doesn't appear in the candidate list though the Mpack is successfully > > > imported. > > > > > > > "In the next version of Bigtop(3.0.1), the bigtop-ambari-mpack is > > > compatible with the version of Ambari(2.7.5, provided by Bigtop > 3.0.1), but > > > the components are provided by Bigtop 1.5.0" > > > > > > To make it true, we have to merge BIGTOP-3591 and BIGTOP-3592 into > > > branch-3.0 (to be released as 3.0.1) in addition to master. > > > Considering that we currently don't have any usable combination of > > > Ambari and Mpack in the 3.0.0 release, we should do that. > > > > > > In addition, I found some problems about Ambari and Mpack on the > > > master branch. I'll file them into JIRA later. > > > > > > Kengo Seki <[email protected]> > > > > > > On Wed, Mar 23, 2022 at 2:07 PM 李帅 <[email protected]> wrote: > > > > > > > > Hi Kengo, > > > > After my testing, I find that the version of Bigtop3.0.0's > > > > bigtop-ambari-mpack is *not* compatible with this version of Ambari > > > > (2.7.5), also is *not *compatible with the previous one (2.6.1, > provided > > > by > > > > Bigtop 1.5.0). > > > > "Ambari 2.7.5 and Mpack built from the master branch" is true. > > > > So, we should note that "In the next version of Bigtop(3.0.1), the > > > > bigtop-ambari-mpack is compatible with the version of Ambari(2.7.5, > > > > provided by Bigtop3.0.1) , but the components are provided by Bigtop > > > 1.5.0" > > > > > > > > peng lee. > > > > > > > > Kengo Seki <[email protected]> 于2022年3月22日周二 07:08写道： > > > > > > > > > Hi Yuqi and Peng, > > > > > > > > > > Sorry for my late response. > > > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 3.0 release > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > For ensuring this, I tried to deploy a cluster using Ambari 2.7.5 > and > > > > > Mpack both provided by Bigtop 3.0.0, > > > > > but unfortunately I came across the following error in the last > step > > > > > of the deployment via Ambari UI. > > > > > > > > > > ``` > > > > > stderr: > > > > > Traceback (most recent call last): > > > > > File > > > > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", > > > > > line 39, in <module> > > > > > AfterInstallHook().execute() > > > > > File > > > > > > > > > "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", > > > > > line 352, in execute > > > > > method(env) > > > > > File > > > > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", > > > > > line 32, in hook > > > > > setup_stack_symlinks(self.stroutfile) > > > > > File > > > > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/shared_initialization.py", > > > > > line 53, in setup_stack_symlinks > > > > > stack_packages = > > > > > stack_select.get_packages(stack_select.PACKAGE_SCOPE_INSTALL) > > > > > File > > > > > > > > > "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", > > > > > line 185, in get_packages > > > > > raise Fail("The stack packages are not defined on the command. > > > > > Unable to load packages for the stack-select tool") > > > > > resource_management.core.exceptions.Fail: The stack packages are > not > > > > > defined on the command. Unable to load packages for the > stack-select > > > > > tool > > > > > ``` > > > > > > > > > > This is because Ambari 2.7.5 and Mpack requires BIGTOP-3592 and > > > > > BIGTOP-3591 respectively to work with each other. > > > > > So I assume Peng meant "Ambari 2.7.5 and Mpack built from the > master > > > > > branch" (not the official 3.0.0 release), > > > > > which these changes were merged into. > > > > > > > > > > Based on the discussion so far, I've updated the Bigtop wiki. Let > me > > > > > know if I misunderstand something. > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+3.0.0+Release > > > > > (diff: > > > > > > > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=191334594&selectedPageVersions=4&selectedPageVersions=3 > > > > > ) > > > > > > > > > > I'll also write a similar note when publishing the 3.0.1 release. > > > > > > > > > > Kengo Seki <[email protected]> > > > > > > > > > > On Fri, Mar 4, 2022 at 11:24 PM 李帅 <[email protected]> > wrote: > > > > > > > > > > > > Oh Kengo and yuqi. It's my fault. I am so sorry about that. > > > Actually > > > > > the > > > > > > Ambari 2.7.5 from Bigtop 3.0 and the Mpack from the Bigtop 3.0, > they > > > are > > > > > > work, but their components are from Bigtop 1.5. So > > > > > > > > > > > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 3.0 release > > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <[email protected]> 于 2022年3月4日周五 08:21写道： > > > > > > > > > > > > > Hi Peng, > > > > > > > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 1.5 release > > > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > > > Thanks! To be precise, the above combination is going to deploy > > > Bigtop > > > > > 1.4 > > > > > > > stack by default > > > > > > > (selecting "Use Public Repository" on Ambari's Web UI shows the > > > > > repository > > > > > > > URL of 1.4.0), > > > > > > > and if users rewrite the repo URL, it can deploy 1.5 stack, > right? > > > > > > > > > > > > > > Kengo Seki <sekikn@ <[email protected]>apache.org> > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 11:13 PM 李帅 <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Hi, Kengo > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 1.5 release > > > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <[email protected]> 于2022年3月3日周四 19:20写道： > > > > > > > > > > > > > > > > > > The bgtp-ambari-mpack is works well with the Ambari 2.7.5 > > > which > > > > > come > > > > > > > > from > > > > > > > > > > Bigtop 3.0, but it's components(Hadoop and Zookeeper) > are all > > > > > come > > > > > > > from > > > > > > > > > Bigtop 1.5. > > > > > > > > > > > > > > > > > > Oh I see, that's what I originally thought so. > > > > > > > > > The mpack of Bigtop 3.0 is intended to deploy Bigtop 1.5 > stack, > > > > > though > > > > > > > > it's > > > > > > > > > a bit misleading :) > > > > > > > > > It should be mentioned in the release notes. > > > > > > > > > > > > > > > > > > Let me organize the combinations of the conditions. My > > > > > understanding is > > > > > > > > as > > > > > > > > > follows, are these correct? > > > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the 3.0.0 > > > release > > > > > > > > > -> It works and deploys Bigtop 1.5 stack. > > > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the upcoming > > > 3.0.1 > > > > > > > release > > > > > > > > > -> Same as the above. > > > > > > > > > (Or, we can merge BIGTOP-3591 into branch-3.0 before > the > > > > > release. > > > > > > > > > In this case, this combination doesn't work) > > > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the > (currently > > > > > > > unreleased) > > > > > > > > > 3.1.0 release > > > > > > > > > -> It doesn't work, since BIGTOP-3591 has already been > > > merged. > > > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the 3.0.0 > > > release > > > > > > > > > -> It doesn't work, since BIGTOP-3591 is not merged yet. > > > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the upcoming > > > 3.0.1 > > > > > > > release > > > > > > > > > -> Same as the above. > > > > > > > > > (Or, we can merge BIGTOP-3591 into branch-3.0 before > the > > > 3.0.1 > > > > > > > > > release. > > > > > > > > > In this case, this combination works and deploys > Bigtop > > > 1.5.0 > > > > > > > > stack) > > > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the > (currently > > > > > > > unreleased) > > > > > > > > > 3.1.0+ release > > > > > > > > > -> It works, and is supposed to deploy Bigtop 3.0 stack > after > > > > > merging > > > > > > > > > Yuqi's upcoming PR. > > > > > > > > > But if we release 3.1.0 based on the current state of > the > > > > > master > > > > > > > > > branch, it deploys Bigtop 1.5.0 stack. > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <[email protected]>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 6:03 PM 李帅 < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Kengo, > > > > > > > > > > I am sorry that my preceding words look ambiguous. > > > > > > > > > > The bgtp-ambari-mpack is works well with the Ambari 2.7.5 > > > which > > > > > come > > > > > > > > from > > > > > > > > > > Bigtop 3.0, but it's components(Hadoop and Zookeeper) > are all > > > > > come > > > > > > > from > > > > > > > > > > Bigtop 1.5. > > > > > > > > > > thanks, > > > > > > > > > > peng lee. > > > > > > > > > > > > > > > > > > > > Kengo Seki <[email protected]> 于 2022年3月3日周四 16:32写道： > > > > > > > > > > > > > > > > > > > > > Thanks for the comment, Yuqi and Lee! (sorry if I > mistook > > > your > > > > > > > name) > > > > > > > > > > > I'll update the release note later. > > > > > > > > > > > > > > > > > > > > > > Let me make sure just one point, Bigtop 1.5 provides > Ambari > > > > > 2.6.1 > > > > > > > > (not > > > > > > > > > > > 2.7.5), > > > > > > > > > > > so I understood "mpack works with Ambari 2.7.5 > installed > > > via > > > > > Bigtop > > > > > > > > > 1.5" > > > > > > > > > > in > > > > > > > > > > > your comments > > > > > > > > > > > is actually "... Ambari 2.6.1 installed via Bigtop > 1.5". > > > > > Correct me > > > > > > > > if > > > > > > > > > > I'm > > > > > > > > > > > wrong :) > > > > > > > > > > > > > > > > > > > > > > > I'd like to start to work on it and add the Mpack > > > Bigtop-3.0 > > > > > > > > support > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > next minor release. > > > > > > > > > > > > > > > > > > > > > > Great, I'm looking forward to it! I'll try it once the > PR > > > is > > > > > > > > submitted. > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <[email protected]>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 2:52 PM 李帅 < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > Actually, bigtop-ambari-mpack provided as a > > > > > > > > > > > > component of Bigtop 3.0 > > > > > > > > > > > > works with Ambari 2.7.5 installed via Bigtop 1.5, > but not > > > > > with > > > > > > > > Ambari > > > > > > > > > > > > 2.7.5/Bigtop 3.0. > > > > > > > > > > > > thanks. > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <[email protected]> 于 2022年3月3日周四 > 07:19写道： > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the comment! > > > > > > > > > > > > > Let me confirm, you mean that bigtop-ambari-mpack > > > which is > > > > > > > > provided > > > > > > > > > > as > > > > > > > > > > > a > > > > > > > > > > > > > component of Bigtop 3.0 > > > > > > > > > > > > > works with Ambari 2.6.1 installed via Bigtop 1.5, > but > > > not > > > > > with > > > > > > > > > Ambari > > > > > > > > > > > > > 2.7.5/Bigtop 3.0, right? > > > > > > > > > > > > > If my understanding is correct, that information is > > > > > significant > > > > > > > > and > > > > > > > > > > I'm > > > > > > > > > > > > > going to reflect it to the release note. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+3.0.0+Release > > > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <[email protected]>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 2, 2022 at 12:48 PM 李帅 < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > hi，Maybe we should note that the > bgtp-ambari-mpack > > > is not > > > > > > > work > > > > > > > > > well > > > > > > > > > > > > with > > > > > > > > > > > > > > bigtop3.x, but it can work well with bigtop 1.5 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <[email protected]> 于 2022年3月1日周二 > > > 21:25写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As discussed in [1], I'd like to publish the > 3.0.1 > > > > > release > > > > > > > > > based > > > > > > > > > > on > > > > > > > > > > > > > > > branch-3.0 soon, > > > > > > > > > > > > > > > which mainly focuses on addressing the > Log4Shell > > > > > > > > > vulnerabilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've created all CI jobs which are required to > that > > > > > > > release, > > > > > > > > > > > > > > > but some of the CI nodes are currently > blacklisted > > > > > from the > > > > > > > > ASF > > > > > > > > > > > > sites. > > > > > > > > > > > > > > > Luca has already asked the ASF Infra to remove > that > > > > > > > > restriction > > > > > > > > > > > [2], > > > > > > > > > > > > > > > but they still seem to be banned as of now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Once they become available again, I'm going to > > > start > > > > > the > > > > > > > > > release > > > > > > > > > > > > > process. > > > > > > > > > > > > > > > (but it will take a few weeks due to long > > > > > > > build/test/transfer > > > > > > > > > > time, > > > > > > > > > > > > as > > > > > > > > > > > > > > > usual) > > > > > > > > > > > > > > > So let me know if there are any fixes that > you'd > > > like > > > > > to > > > > > > > > > include > > > > > > > > > > > into > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/vor809oqqcjxso98nxrml5s3lthrlws5 > > > > > > > > > > > > > > > [2]: > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/BIGTOP-3644#comment-17498543 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <[email protected]> > apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
