Hi community, In my original idea, I was planning to provide Bigtop3.1.0 + Ambari2.7.5 for users at first, and keep all big changes in Ambari2.8.0, e.g. 1、Fix some security problems by upgrading Spring Framework and others 2、Frontend refactor 3、Upgrade python version etc.
All these changes will take long time, which may be bad for the release cycle of Ambari itself. So I'm considering we can move Mpack development to Ambari side, and 2.8.0 is all about Bigtop stack, move other big changes to next major version(except the security problem, which absolutely need to be fixed in next release). But we need to make final decision for the things we discussed before, like, 1、Add prefix to installation path(/usr/bigtop) 2、Which repo to store the packages(with prefix in installation path)? 3、Related name(Stack name: BIGTOP, scripts: conf-select, distro-select, bigtop-select are all ok for me)? Any other things I forgot? What’s your opinion? Best Regards, Zhiguo Wu On 2022/06/23 06:26:44 "Battula, Brahma Reddy" wrote: > > IMO, First way is not heavier when we consider all the efforts for mpack with > upgrades. And first one is easiest to adopt now. > > We'll start discussing the bigtop mailing list further. > > > On 21/06/22, 1:11 PM, "吴治国" <[email protected]> wrote: > > Thanks for the details, Brahma and Kengo. > > If I understand it correctly, there are two approaches under discussion: > > 1、Prepare to release 2.7.7, in this version, we won't use HDP components > anymore, we'll replace it with the community version provided by Bigtop, and > update RPM and DEB build scripts provided by Bigtop, change the installation > path so hdp-select can be reused. (Due to component version changes, I don't > know if the code under /stacks needs to be updated) > 2、Develop Bigtop mpack for Ambari 2.7.5, we don't need to change Ambari's > code (if the code under /stacks can be reused in the first way, maybe it also > works in this way?) > > Isn't the workload of the first way heavier? > > > About the meeting > > I thought maybe email discussion would be better? > The discussion can be open for a few days and all interested people in > the community can participate, not just a few of us. > And due to time zone and language issues, many people are hard to > attending the meetings, email discussions may be more suitable for these > people. > > What do you think? > > Best Regards, > Zhiguo Wu > > > On Jun 20, 2022, at 23:15, Battula, Brahma Reddy > <[email protected]> wrote: > > > >>> Assuming the understanding above is correct, my new concerns are: > > > > Yes, Hope we'll in same page now. Still we can discuss in weekly call. > > > >>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's > trademark? > >>> If so, should we give a new stack name to it, as you mentioned as > >>> planA in [3]? > >>> (or it makes upgrading HDP cluster difficult? I don't understand it > so much) > > > > Yes, it may be. Hence I introduced the planA and preferred it. one more > idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not > sure whether it should be ok or not(and we might not delivering as distro's). > > > >>> * It may be a bit tough work to fix Bigtop's build scripts so that its > >>> artifacts match Ambari's requirements. > >>> (package versioning, file layouts, included and not included files, > >>> file permissions, user/group creation, etc.) > > > > Only paths matter right everything else can be re-used from the > hdp-select. > > > > > > On 20/06/22, 8:13 PM, "Kengo Seki" <[email protected] > <[email protected]>> wrote: > > > > Thanks for the elaboration Brahma, but I'm a bit confused. > > At this point, my understanding is as follows. Are these correct? > > > > * In the 2.7.x line, we'll still use a stack definition based on HDP's > one [1]. > > > > * But regarding RPM and DEB packages, we'll build them from source by > leveraging > > Bigtop's build scripts and publish them on the ASF distribution site > > or somewhere. > > repoinfo.xml [2] will also be fixed to point that repository. > > > > Assuming the understanding above is correct, my new concerns are: > > > > * Can we still call it as "HDP"? Doesn't it infringe Cloudera's > trademark? > > If so, should we give a new stack name to it, as you mentioned as > > planA in [3]? > > (or it makes upgrading HDP cluster difficult? I don't understand it so > much) > > > > * It may be a bit tough work to fix Bigtop's build scripts so that its > > artifacts match Ambari's requirements. > > (package versioning, file layouts, included and not included files, > > file permissions, user/group creation, etc.) > > I'm not sure if it's faster than developing Mpack. > > > > [1]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dFX07wfIY8aukfEUcJe9GN0rds8t3%2Fpenmsiszglkn4%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dFX07wfIY8aukfEUcJe9GN0rds8t3%2Fpenmsiszglkn4%3D&reserved=0> > > [2]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MDz57IloHfWYWcwmAI9h1pYDG45v1UbU6K2M%2BkpcDEA%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MDz57IloHfWYWcwmAI9h1pYDG45v1UbU6K2M%2BkpcDEA%3D&reserved=0> > > [3]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=P1pPHwM1YggdPT9qD4aAvJdehjjZdZ8mLhvn0ZfR7Gw%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=P1pPHwM1YggdPT9qD4aAvJdehjjZdZ8mLhvn0ZfR7Gw%3D&reserved=0> > > > > Kengo Seki <[email protected] <[email protected]>> > > > > On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy > > <[email protected] <[email protected]>> wrote: > >> > >> We'll not use the HDP Packages (Only selects we'll use from hdp, which > can be plugged in bigtop). Still we'll use opensource packages and bigtop to > create the RPM's and selects. > >> Just we can use the source tar balls opensource components(e.g > Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in > following repo. Or from the apache github repo's. > >> > >> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyNyy7fOUlWBSFMJdd1iRMrmlKOVyA5NEGuMpSkCSEs%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RyNyy7fOUlWBSFMJdd1iRMrmlKOVyA5NEGuMpSkCSEs%3D&reserved=0> > >> > >> Planning to have sync up call on this week, once it's finalize, we can > summarise here the approach. What do you think..? > >> > >> > >> On 20/06/22, 4:06 PM, "Kengo Seki" <[email protected] > <[email protected]>> wrote: > >> > >> Thanks for the reply, Brahma! > >> > >>> We'll use following repo, how other components in Apache are hosted. > >> > >> Using that site for distribution has no problem. My concern is, do we > >> have the source tree for building HDP packages and does it still work > >> without the artifacts behind the paywall? > >> According to ASF's release policy [1], we must publish the > >> corresponding source releases whenever we publish binary artifacts, > >> such as RPM and DEB packages for HDP. > >> > >> [1]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPEbEEpIN8NCsrMnxW5xKbB%2B19bDJprq%2FiXjIF09vls%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qPEbEEpIN8NCsrMnxW5xKbB%2B19bDJprq%2FiXjIF09vls%3D&reserved=0> > >> > >> Kengo Seki <[email protected] <[email protected]>> > >> > >> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy > >> <[email protected] <[email protected]>> wrote: > >>> > >>> Hi Kengo Seki, > >>> > >>> Thanks for pitching here.. We'll use following repo, how other > components in Apache are hosted. Any other suggestions..? > >>> > >>> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OUGhE45KsDvPum4%2Fd9cwlJPfH66OlKE%2BgD557%2BiiAC8%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OUGhE45KsDvPum4%2Fd9cwlJPfH66OlKE%2BgD557%2BiiAC8%3D&reserved=0> > >>> > >>> > >>> > >>> On 20/06/22, 2:32 PM, "Kengo Seki" <[email protected] > <[email protected]>> wrote: > >>> > >>> Oh, I've just noticed that I submitted an additional question into the > >>> wrong thread mistakenly. > >>> > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UTDhFu1Bloa3FzD0shWWHoDt%2FrQD22JziV1%2BNAWp8NQ%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UTDhFu1Bloa3FzD0shWWHoDt%2FrQD22JziV1%2BNAWp8NQ%3D&reserved=0> > is > >>> originally intended as a reply to this thread. Sorry for the > >>> confusion. > >>> > >>> Kengo Seki <[email protected] <[email protected]>> > >>> > >>> On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <[email protected] > <[email protected]>> wrote: > >>>> > >>>> Thank you so much for writing the draft and commenting on it, Brahma > and Viraj! > >>>> Firstly, a minor comment: > >>>> > >>>>> Trunk Code : Parallel we can work for the trunk code, this we might > give as 2.8. > >>>>> * Working on bigtop mpack > >>>> > >>>> This means adding a new built-in stack which uses the Bigtop > >>>> repository, and not developing Mpack on the Bigtop side, right? > >>>> If so, "bigtop mpack" sounds confusing a bit. > >>>> > >>>>> hdp-select can stay on branch-2.7 and it can be moved to bigtop for > trunk branch. Thoughts? > >>>> > >>>> Sounds good to me. During the 2.7.x releases, users can use the HDP > >>>> stack by default. In addition, they can also add the Bigtop stack > >>>> through the Mpack, as Zhiguo mentioned in [1]. > >>>> Since 2.8 or 3.0, we will drop HDP and users can use the built-in > >>>> Bigtop stack by default. It also allows us to drop the Mpack on the > >>>> Bigtop side. > >>>> > >>>> Regarding the built-in Bigtop stack developed on trunk, the current > >>>> BIGTOP stack in Ambari [2] may be a good start point to be upgraded. > >>>> I ensured that Ambari 2.7.6 was successfully built with the > >>>> `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to > >>>> use for now. > >>>> > >>>> [1]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iqQ6QCM1%2BMYJ61tU%2FFk%2BWC8GvftYYGhrnu6V%2FFJ3d7o%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iqQ6QCM1%2BMYJ61tU%2FFk%2BWC8GvftYYGhrnu6V%2FFJ3d7o%3D&reserved=0> > >>>> [2]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FR5NTHCZt%2FQcqy9NyT63Q3NIYlEeJ6sIpbOqzfDOMRY%3D&reserved=0 > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Cf26dc731668c471ef20008da53597102%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913940886207699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi [message truncated...]
