Thanks for the elaboration Brahma, but I'm a bit confused. At this point, my understanding is as follows. Are these correct?
* In the 2.7.x line, we'll still use a stack definition based on HDP's one [1]. * But regarding RPM and DEB packages, we'll build them from source by leveraging Bigtop's build scripts and publish them on the ASF distribution site or somewhere. repoinfo.xml [2] will also be fixed to point that repository. Assuming the understanding above is correct, my new concerns are: * Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark? If so, should we give a new stack name to it, as you mentioned as planA in [3]? (or it makes upgrading HDP cluster difficult? I don't understand it so much) * It may be a bit tough work to fix Bigtop's build scripts so that its artifacts match Ambari's requirements. (package versioning, file layouts, included and not included files, file permissions, user/group creation, etc.) I'm not sure if it's faster than developing Mpack. [1]: https://github.com/apache/ambari/tree/release-2.7.6/ambari-server/src/main/resources/stacks/HDP [2]: https://github.com/apache/ambari/blob/release-2.7.6/ambari-server/src/main/resources/stacks/HDP/2.6/repos/repoinfo.xml [3]: https://lists.apache.org/thread/m77lhzo0njr2dhoock5twtm0c19j2py8 Kengo Seki <[email protected]> On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy <[email protected]> wrote: > > We'll not use the HDP Packages (Only selects we'll use from hdp, which can be > plugged in bigtop). Still we'll use opensource packages and bigtop to create > the RPM's and selects. > Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 > source tarballs from apache repo) and build rpm's and publish in following > repo. Or from the apache github repo's. > > https://dlcdn.apache.org/bigtop/bigtop-3.1.0/repos/ > > Planning to have sync up call on this week, once it's finalize, we can > summarise here the approach. What do you think..? > > > On 20/06/22, 4:06 PM, "Kengo Seki" <[email protected]> wrote: > > Thanks for the reply, Brahma! > > > We'll use following repo, how other components in Apache are hosted. > > Using that site for distribution has no problem. My concern is, do we > have the source tree for building HDP packages and does it still work > without the artifacts behind the paywall? > According to ASF's release policy [1], we must publish the > corresponding source releases whenever we publish binary artifacts, > such as RPM and DEB packages for HDP. > > [1]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qwRVqj4GzMqB%2BfeVfzMemWJIEjDHz5okUrYXdH4vj00%3D&reserved=0 > > Kengo Seki <[email protected]> > > On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy > <[email protected]> wrote: > > > > Hi Kengo Seki, > > > > Thanks for pitching here.. We'll use following repo, how other > components in Apache are hosted. Any other suggestions..? > > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FWqjRiYl0WOptyA5P42xwP5S6KHiK1QrIi5avv1vtHw%3D&reserved=0 > > > > > > > > On 20/06/22, 2:32 PM, "Kengo Seki" <[email protected]> wrote: > > > > Oh, I've just noticed that I submitted an additional question into > the > > wrong thread mistakenly. > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JxCXNEbqlknyBZES5PoFrk356zlDeO1uUtGGl677kqk%3D&reserved=0 > is > > originally intended as a reply to this thread. Sorry for the > > confusion. > > > > Kengo Seki <[email protected]> > > > > On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <[email protected]> > wrote: > > > > > > Thank you so much for writing the draft and commenting on it, > Brahma and Viraj! > > > Firstly, a minor comment: > > > > > > > Trunk Code : Parallel we can work for the trunk code, this we > might give as 2.8. > > > > * Working on bigtop mpack > > > > > > This means adding a new built-in stack which uses the Bigtop > > > repository, and not developing Mpack on the Bigtop side, right? > > > If so, "bigtop mpack" sounds confusing a bit. > > > > > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop > for trunk branch. Thoughts? > > > > > > Sounds good to me. During the 2.7.x releases, users can use the > HDP > > > stack by default. In addition, they can also add the Bigtop stack > > > through the Mpack, as Zhiguo mentioned in [1]. > > > Since 2.8 or 3.0, we will drop HDP and users can use the built-in > > > Bigtop stack by default. It also allows us to drop the Mpack on > the > > > Bigtop side. > > > > > > Regarding the built-in Bigtop stack developed on trunk, the > current > > > BIGTOP stack in Ambari [2] may be a good start point to be > upgraded. > > > I ensured that Ambari 2.7.6 was successfully built with the > > > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too > old to > > > use for now. > > > > > > [1]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5nWc%2BIgBeDof0a0wdth4X0JrGPNaUkse6ye66XpNFiM%3D&reserved=0 > > > [2]: > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ExDSC%2BX%2BIhqdt8ok8w5jxFE2vDAlIiOQRmFkBXZ4XwY%3D&reserved=0 > > > > > > Kengo Seki <[email protected]> > > > > > > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <[email protected]> > wrote: > > > > > > > > Thanks for the nice draft, Brahma! > > > > > > > > IMHO, as we narrow down to 2.7.7 release, it might be still > worth sticking > > > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. > For 2.8.0 > > > > (or 3.0.0), we can add entire new stack (BDP sounds good one) > and start off > > > > with 2.8.0/3.0.0 stack version. > > > > > > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop > for trunk > > > > branch. Thoughts? > > > > > > > > Similarly, we can start-off with "python 3 only" support in > trunk, whereas > > > > 2.7 release line can continue with python 2 to reduce > complexities for > > > > maintenance releases on 2.7. > > > > > > > > All other points look good. Once again, thank you for drafting > the roadmap > > > > and starting with this thread. > > > > > > > > > > > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy > > > > <[email protected]> wrote: > > > > > > > > > Hi All, > > > > > > > > > > As Ambari back now, let’s start discussing roadmap. I will be > closely > > > > > tracking for Jira, GitHub(with Jenkins) and wiki to work > normal. > > > > > Based on the conclusion of this thread will update wiki[1]. > > > > > > > > > > As 2.7.6 is released recently, we need to release 2.7.7 > > > > > > > > > > > > > > > 1. Stack Management. There are two approaches for this. > Open to discuss > > > > > on following.. > > > > > * Entirely new stack > > > > > > > > > > > i. > > > > > Define new stack name. (BDP: BigDataPlatform, > BDS:BigDataStack, > > > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…) > > > > > > > > > > > ii. > > > > > Define stack versions > > > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..) > > > > > > > > > > > iii. Based > > > > > above two we need to have select in bigtop code. > > > > > > > > > > * Using the existing for current release. > > > > > > > > > > > i. > > > > > Plugin the new stack into HDP self > > > > > > > > > > > ii. > > > > > Still, we can have hdp select > > > > > > > > > > > > > > > > > > > > I prefer the planA but planB takes less time. Users can > easily adopt it > > > > > upgrade their stack. > > > > > > > > > > > > > > > > > > > > Note: Ideally bigtop mpack can be long term approach as stack > id decoupled > > > > > now, but currently it’s not ready for latest stack (upgrades > can be risk). > > > > > We’ll work in trunk for same. If this works shortly, we can > choose. > > > > > > > > > > > > > > > > > > > > 1. Upgrade : > > > > > * Most of users will be deployed with HDP > > > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the > upgrade to 2.7.7 > > > > > 2. Other work > > > > > * Any critical/blocker CVE’s Fixes > > > > > * CVEs Fixes : we’ll try to target some CVE fixes > > > > > * Review the open PR’s and Jira’s > > > > > > > > > > > > > > > > > > > > Trunk Code : Parallel we can work for the trunk code, this we > might give > > > > > as 2.8. > > > > > > > > > > * Removal of python2 support > > > > > * Removal of hard dependency on HDP > > > > > * Working on bigtop mpack > > > > > > > > > > Can add more based on your inputs… > > > > > > > > > > > > > > > Any thoughts on this..? Please feel free to add anything > which I missed. > > > > > > > > > > > > > > > > > > > > Reference: > > > > > > > > > > 1. > > > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s%2FI1zyAlaiuvWqx%2FjZo%2FNBu4KeHrPVmxtIfeYgLQAso%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
