Hi Michiel, I really appreciate your offer, that sounds very useful for the HDP users. But to be honest, I'm afraid I'm a bit reluctant to include it into Bigtop, due to the following reasons.
1. Ambari has already been retired this January [1]. Keeping retired software as a Bigtop component brings us security risks and maintenance cost. Especially, Ambari depends on several obsolete softwares such as Python2 and Bower. So I personally think we should drop Ambari (and also Mpack) at some point in the future, unless it revives. 2. Cloudera has changed the license of their product [2], so I'm not sure if we could do that. Doesn't it violate their license to install and use HDP without subscription? If the new Mpack is derived from their product, could we include it in our distributions from the viewpoint of license compatibility [3]? But the above is just my personal opinion, so I'd like to hear from others. [1]: https://lists.apache.org/thread/m5jrpn4j28kn3wfn4zzxvy0g450vdlr1 [2]: https://www.cloudera.com/downloads/paywall-expansion.html [3]: https://www.apache.org/legal/resolved.html Kengo Seki <sek...@apache.org> On Sat, Mar 26, 2022 at 4:40 PM Michiel Verheul <chee...@gmail.com> wrote: > > Hi Kengo, > > I'm currently working on porting the Hortonworks HDP 3.1 mpack to make it > compatible with Bigtop 3.0.0. > Do you think it's an idea to include that as a separate (BGTP-3.0) mpack in > the bigtop repo? I've just reached the point that it installs from Ambari, > completes the setup wizard and runs a > HDFS+YARN+Mapreduce2+Hive+Zookeeper+Spark cluster without any errors. But a > lot more testing has to be done to see if all services run nicely. > Apart from that, I also want to enable the rest of the HDP Services that > are also included in Bigtop. (HBase, Kafka, Flink, Zeppelin, etc) > Please let me know what you think. > > > Op za 26 mrt. 2022 07:38 schreef Kengo Seki <sek...@apache.org>: > > > Hi Peng, > > > > Thank you so much for testing! > > > > > I find that the version of Bigtop3.0.0's bigtop-ambari-mpack is *not* > > compatible with this version of Ambari (2.7.5), also is *not* compatible > > with the previous one (2.6.1, provided by Bigtop 1.5.0). > > > > I also confirmed that combination doesn't work. The Bigtop stack > > doesn't appear in the candidate list though the Mpack is successfully > > imported. > > > > > "In the next version of Bigtop(3.0.1), the bigtop-ambari-mpack is > > compatible with the version of Ambari(2.7.5, provided by Bigtop 3.0.1), but > > the components are provided by Bigtop 1.5.0" > > > > To make it true, we have to merge BIGTOP-3591 and BIGTOP-3592 into > > branch-3.0 (to be released as 3.0.1) in addition to master. > > Considering that we currently don't have any usable combination of > > Ambari and Mpack in the 3.0.0 release, we should do that. > > > > In addition, I found some problems about Ambari and Mpack on the > > master branch. I'll file them into JIRA later. > > > > Kengo Seki <sek...@apache.org> > > > > On Wed, Mar 23, 2022 at 2:07 PM 李帅 <lishuaipeng...@gmail.com> wrote: > > > > > > Hi Kengo, > > > After my testing, I find that the version of Bigtop3.0.0's > > > bigtop-ambari-mpack is *not* compatible with this version of Ambari > > > (2.7.5), also is *not *compatible with the previous one (2.6.1, provided > > by > > > Bigtop 1.5.0). > > > "Ambari 2.7.5 and Mpack built from the master branch" is true. > > > So, we should note that "In the next version of Bigtop(3.0.1), the > > > bigtop-ambari-mpack is compatible with the version of Ambari(2.7.5, > > > provided by Bigtop3.0.1) , but the components are provided by Bigtop > > 1.5.0" > > > > > > peng lee. > > > > > > Kengo Seki <sek...@apache.org> 于2022年3月22日周二 07:08写道： > > > > > > > Hi Yuqi and Peng, > > > > > > > > Sorry for my late response. > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 3.0 release > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > For ensuring this, I tried to deploy a cluster using Ambari 2.7.5 and > > > > Mpack both provided by Bigtop 3.0.0, > > > > but unfortunately I came across the following error in the last step > > > > of the deployment via Ambari UI. > > > > > > > > ``` > > > > stderr: > > > > Traceback (most recent call last): > > > > File > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", > > > > line 39, in <module> > > > > AfterInstallHook().execute() > > > > File > > > > > > "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", > > > > line 352, in execute > > > > method(env) > > > > File > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/hook.py", > > > > line 32, in hook > > > > setup_stack_symlinks(self.stroutfile) > > > > File > > > > > > "/var/lib/ambari-agent/cache/stack-hooks/after-INSTALL/scripts/shared_initialization.py", > > > > line 53, in setup_stack_symlinks > > > > stack_packages = > > > > stack_select.get_packages(stack_select.PACKAGE_SCOPE_INSTALL) > > > > File > > > > > > "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/stack_select.py", > > > > line 185, in get_packages > > > > raise Fail("The stack packages are not defined on the command. > > > > Unable to load packages for the stack-select tool") > > > > resource_management.core.exceptions.Fail: The stack packages are not > > > > defined on the command. Unable to load packages for the stack-select > > > > tool > > > > ``` > > > > > > > > This is because Ambari 2.7.5 and Mpack requires BIGTOP-3592 and > > > > BIGTOP-3591 respectively to work with each other. > > > > So I assume Peng meant "Ambari 2.7.5 and Mpack built from the master > > > > branch" (not the official 3.0.0 release), > > > > which these changes were merged into. > > > > > > > > Based on the discussion so far, I've updated the Bigtop wiki. Let me > > > > know if I misunderstand something. > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+3.0.0+Release > > > > (diff: > > > > > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=191334594&selectedPageVersions=4&selectedPageVersions=3 > > > > ) > > > > > > > > I'll also write a similar note when publishing the 3.0.1 release. > > > > > > > > Kengo Seki <sek...@apache.org> > > > > > > > > On Fri, Mar 4, 2022 at 11:24 PM 李帅 <lishuaipeng...@gmail.com> wrote: > > > > > > > > > > Oh Kengo and yuqi. It's my fault. I am so sorry about that. > > Actually > > > > the > > > > > Ambari 2.7.5 from Bigtop 3.0 and the Mpack from the Bigtop 3.0, they > > are > > > > > work, but their components are from Bigtop 1.5. So > > > > > > > > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 3.0 release > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > > > > > > > > > Kengo Seki <sek...@apache.org> 于 2022年3月4日周五 08:21写道： > > > > > > > > > > > Hi Peng, > > > > > > > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 1.5 release > > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > Thanks! To be precise, the above combination is going to deploy > > Bigtop > > > > 1.4 > > > > > > stack by default > > > > > > (selecting "Use Public Repository" on Ambari's Web UI shows the > > > > repository > > > > > > URL of 1.4.0), > > > > > > and if users rewrite the repo URL, it can deploy 1.5 stack, right? > > > > > > > > > > > > Kengo Seki <sekikn@ <sek...@gmail.com>apache.org> > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 11:13 PM 李帅 <lishuaipeng...@gmail.com> > > wrote: > > > > > > > > > > > > > Hi, Kengo > > > > > > > * Ambari 2.7.5(from Bigtop 3.0) w/ Mpack from the 1.5 release > > > > > > > -> it works and deploys Bigtop 1.5 stack > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sek...@apache.org> 于2022年3月3日周四 19:20写道： > > > > > > > > > > > > > > > > The bgtp-ambari-mpack is works well with the Ambari 2.7.5 > > which > > > > come > > > > > > > from > > > > > > > > > Bigtop 3.0, but it's components(Hadoop and Zookeeper) are all > > > > come > > > > > > from > > > > > > > > Bigtop 1.5. > > > > > > > > > > > > > > > > Oh I see, that's what I originally thought so. > > > > > > > > The mpack of Bigtop 3.0 is intended to deploy Bigtop 1.5 stack, > > > > though > > > > > > > it's > > > > > > > > a bit misleading :) > > > > > > > > It should be mentioned in the release notes. > > > > > > > > > > > > > > > > Let me organize the combinations of the conditions. My > > > > understanding is > > > > > > > as > > > > > > > > follows, are these correct? > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the 3.0.0 > > release > > > > > > > > -> It works and deploys Bigtop 1.5 stack. > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the upcoming > > 3.0.1 > > > > > > release > > > > > > > > -> Same as the above. > > > > > > > > (Or, we can merge BIGTOP-3591 into branch-3.0 before the > > > > release. > > > > > > > > In this case, this combination doesn't work) > > > > > > > > > > > > > > > > * Ambari 2.6.1 (from Bigtop 1.5) w/ Mpack from the (currently > > > > > > unreleased) > > > > > > > > 3.1.0 release > > > > > > > > -> It doesn't work, since BIGTOP-3591 has already been > > merged. > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the 3.0.0 > > release > > > > > > > > -> It doesn't work, since BIGTOP-3591 is not merged yet. > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the upcoming > > 3.0.1 > > > > > > release > > > > > > > > -> Same as the above. > > > > > > > > (Or, we can merge BIGTOP-3591 into branch-3.0 before the > > 3.0.1 > > > > > > > > release. > > > > > > > > In this case, this combination works and deploys Bigtop > > 1.5.0 > > > > > > > stack) > > > > > > > > > > > > > > > > * Ambari 2.7.5 (from Bigtop 3.0) w/ Mpack from the (currently > > > > > > unreleased) > > > > > > > > 3.1.0+ release > > > > > > > > -> It works, and is supposed to deploy Bigtop 3.0 stack after > > > > merging > > > > > > > > Yuqi's upcoming PR. > > > > > > > > But if we release 3.1.0 based on the current state of the > > > > master > > > > > > > > branch, it deploys Bigtop 1.5.0 stack. > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <sek...@gmail.com>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 6:03 PM 李帅 <lishuaipeng...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > Hi Kengo, > > > > > > > > > I am sorry that my preceding words look ambiguous. > > > > > > > > > The bgtp-ambari-mpack is works well with the Ambari 2.7.5 > > which > > > > come > > > > > > > from > > > > > > > > > Bigtop 3.0, but it's components(Hadoop and Zookeeper) are all > > > > come > > > > > > from > > > > > > > > > Bigtop 1.5. > > > > > > > > > thanks, > > > > > > > > > peng lee. > > > > > > > > > > > > > > > > > > Kengo Seki <sek...@apache.org> 于 2022年3月3日周四 16:32写道： > > > > > > > > > > > > > > > > > > > Thanks for the comment, Yuqi and Lee! (sorry if I mistook > > your > > > > > > name) > > > > > > > > > > I'll update the release note later. > > > > > > > > > > > > > > > > > > > > Let me make sure just one point, Bigtop 1.5 provides Ambari > > > > 2.6.1 > > > > > > > (not > > > > > > > > > > 2.7.5), > > > > > > > > > > so I understood "mpack works with Ambari 2.7.5 installed > > via > > > > Bigtop > > > > > > > > 1.5" > > > > > > > > > in > > > > > > > > > > your comments > > > > > > > > > > is actually "... Ambari 2.6.1 installed via Bigtop 1.5". > > > > Correct me > > > > > > > if > > > > > > > > > I'm > > > > > > > > > > wrong :) > > > > > > > > > > > > > > > > > > > > > I'd like to start to work on it and add the Mpack > > Bigtop-3.0 > > > > > > > support > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > next minor release. > > > > > > > > > > > > > > > > > > > > Great, I'm looking forward to it! I'll try it once the PR > > is > > > > > > > submitted. > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <sek...@gmail.com>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 3, 2022 at 2:52 PM 李帅 < > > lishuaipeng...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > Actually, bigtop-ambari-mpack provided as a > > > > > > > > > > > component of Bigtop 3.0 > > > > > > > > > > > works with Ambari 2.7.5 installed via Bigtop 1.5, but not > > > > with > > > > > > > Ambari > > > > > > > > > > > 2.7.5/Bigtop 3.0. > > > > > > > > > > > thanks. > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sek...@apache.org> 于 2022年3月3日周四 07:19写道： > > > > > > > > > > > > > > > > > > > > > > > Thank you for the comment! > > > > > > > > > > > > Let me confirm, you mean that bigtop-ambari-mpack > > which is > > > > > > > provided > > > > > > > > > as > > > > > > > > > > a > > > > > > > > > > > > component of Bigtop 3.0 > > > > > > > > > > > > works with Ambari 2.6.1 installed via Bigtop 1.5, but > > not > > > > with > > > > > > > > Ambari > > > > > > > > > > > > 2.7.5/Bigtop 3.0, right? > > > > > > > > > > > > If my understanding is correct, that information is > > > > significant > > > > > > > and > > > > > > > > > I'm > > > > > > > > > > > > going to reflect it to the release note. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+3.0.0+Release > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <sek...@gmail.com>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 2, 2022 at 12:48 PM 李帅 < > > > > lishuaipeng...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > hi，Maybe we should note that the bgtp-ambari-mpack > > is not > > > > > > work > > > > > > > > well > > > > > > > > > > > with > > > > > > > > > > > > > bigtop3.x, but it can work well with bigtop 1.5 > > > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sek...@apache.org> 于 2022年3月1日周二 > > 21:25写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > As discussed in [1], I'd like to publish the 3.0.1 > > > > release > > > > > > > > based > > > > > > > > > on > > > > > > > > > > > > > > branch-3.0 soon, > > > > > > > > > > > > > > which mainly focuses on addressing the Log4Shell > > > > > > > > vulnerabilities. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've created all CI jobs which are required to that > > > > > > release, > > > > > > > > > > > > > > but some of the CI nodes are currently blacklisted > > > > from the > > > > > > > ASF > > > > > > > > > > > sites. > > > > > > > > > > > > > > Luca has already asked the ASF Infra to remove that > > > > > > > restriction > > > > > > > > > > [2], > > > > > > > > > > > > > > but they still seem to be banned as of now. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Once they become available again, I'm going to > > start > > > > the > > > > > > > > release > > > > > > > > > > > > process. > > > > > > > > > > > > > > (but it will take a few weeks due to long > > > > > > build/test/transfer > > > > > > > > > time, > > > > > > > > > > > as > > > > > > > > > > > > > > usual) > > > > > > > > > > > > > > So let me know if there are any fixes that you'd > > like > > > > to > > > > > > > > include > > > > > > > > > > into > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > https://lists.apache.org/thread/vor809oqqcjxso98nxrml5s3lthrlws5 > > > > > > > > > > > > > > [2]: > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/BIGTOP-3644#comment-17498543 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kengo Seki <sekikn@ <sek...@gmail.com>apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
