We'll not use the HDP Packages (Only selects we'll use from hdp, which can be plugged in bigtop). Still we'll use opensource packages and bigtop to create the RPM's and selects. Just we can use the source tar balls opensource components(e.g Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in following repo. Or from the apache github repo's.
https://dlcdn.apache.org/bigtop/bigtop-3.1.0/repos/ Planning to have sync up call on this week, once it's finalize, we can summarise here the approach. What do you think..? On 20/06/22, 4:06 PM, "Kengo Seki" <[email protected]> wrote: Thanks for the reply, Brahma! > We'll use following repo, how other components in Apache are hosted. Using that site for distribution has no problem. My concern is, do we have the source tree for building HDP packages and does it still work without the artifacts behind the paywall? According to ASF's release policy [1], we must publish the corresponding source releases whenever we publish binary artifacts, such as RPM and DEB packages for HDP. [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qwRVqj4GzMqB%2BfeVfzMemWJIEjDHz5okUrYXdH4vj00%3D&reserved=0 Kengo Seki <[email protected]> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy <[email protected]> wrote: > > Hi Kengo Seki, > > Thanks for pitching here.. We'll use following repo, how other components in Apache are hosted. Any other suggestions..? > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FWqjRiYl0WOptyA5P42xwP5S6KHiK1QrIi5avv1vtHw%3D&reserved=0 > > > > On 20/06/22, 2:32 PM, "Kengo Seki" <[email protected]> wrote: > > Oh, I've just noticed that I submitted an additional question into the > wrong thread mistakenly. > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JxCXNEbqlknyBZES5PoFrk356zlDeO1uUtGGl677kqk%3D&reserved=0 is > originally intended as a reply to this thread. Sorry for the > confusion. > > Kengo Seki <[email protected]> > > On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <[email protected]> wrote: > > > > Thank you so much for writing the draft and commenting on it, Brahma and Viraj! > > Firstly, a minor comment: > > > > > Trunk Code : Parallel we can work for the trunk code, this we might give as 2.8. > > > * Working on bigtop mpack > > > > This means adding a new built-in stack which uses the Bigtop > > repository, and not developing Mpack on the Bigtop side, right? > > If so, "bigtop mpack" sounds confusing a bit. > > > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk branch. Thoughts? > > > > Sounds good to me. During the 2.7.x releases, users can use the HDP > > stack by default. In addition, they can also add the Bigtop stack > > through the Mpack, as Zhiguo mentioned in [1]. > > Since 2.8 or 3.0, we will drop HDP and users can use the built-in > > Bigtop stack by default. It also allows us to drop the Mpack on the > > Bigtop side. > > > > Regarding the built-in Bigtop stack developed on trunk, the current > > BIGTOP stack in Ambari [2] may be a good start point to be upgraded. > > I ensured that Ambari 2.7.6 was successfully built with the > > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is too old to > > use for now. > > > > [1]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5nWc%2BIgBeDof0a0wdth4X0JrGPNaUkse6ye66XpNFiM%3D&reserved=0 > > [2]: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ExDSC%2BX%2BIhqdt8ok8w5jxFE2vDAlIiOQRmFkBXZ4XwY%3D&reserved=0 > > > > Kengo Seki <[email protected]> > > > > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani <[email protected]> wrote: > > > > > > Thanks for the nice draft, Brahma! > > > > > > IMHO, as we narrow down to 2.7.7 release, it might be still worth sticking > > > to HDP i.e. plugin the new stack for HDP for 2.7 release line. For 2.8.0 > > > (or 3.0.0), we can add entire new stack (BDP sounds good one) and start off > > > with 2.8.0/3.0.0 stack version. > > > > > > hdp-select can stay on branch-2.7 and it can be moved to bigtop for trunk > > > branch. Thoughts? > > > > > > Similarly, we can start-off with "python 3 only" support in trunk, whereas > > > 2.7 release line can continue with python 2 to reduce complexities for > > > maintenance releases on 2.7. > > > > > > All other points look good. Once again, thank you for drafting the roadmap > > > and starting with this thread. > > > > > > > > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy > > > <[email protected]> wrote: > > > > > > > Hi All, > > > > > > > > As Ambari back now, let’s start discussing roadmap. I will be closely > > > > tracking for Jira, GitHub(with Jenkins) and wiki to work normal. > > > > Based on the conclusion of this thread will update wiki[1]. > > > > > > > > As 2.7.6 is released recently, we need to release 2.7.7 > > > > > > > > > > > > 1. Stack Management. There are two approaches for this. Open to discuss > > > > on following.. > > > > * Entirely new stack > > > > > > > > i. > > > > Define new stack name. (BDP: BigDataPlatform, BDS:BigDataStack, > > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…) > > > > > > > > ii. > > > > Define stack versions > > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..) > > > > > > > > iii. Based > > > > above two we need to have select in bigtop code. > > > > > > > > * Using the existing for current release. > > > > > > > > i. > > > > Plugin the new stack into HDP self > > > > > > > > ii. > > > > Still, we can have hdp select > > > > > > > > > > > > > > > > I prefer the planA but planB takes less time. Users can easily adopt it > > > > upgrade their stack. > > > > > > > > > > > > > > > > Note: Ideally bigtop mpack can be long term approach as stack id decoupled > > > > now, but currently it’s not ready for latest stack (upgrades can be risk). > > > > We’ll work in trunk for same. If this works shortly, we can choose. > > > > > > > > > > > > > > > > 1. Upgrade : > > > > * Most of users will be deployed with HDP > > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support the upgrade to 2.7.7 > > > > 2. Other work > > > > * Any critical/blocker CVE’s Fixes > > > > * CVEs Fixes : we’ll try to target some CVE fixes > > > > * Review the open PR’s and Jira’s > > > > > > > > > > > > > > > > Trunk Code : Parallel we can work for the trunk code, this we might give > > > > as 2.8. > > > > > > > > * Removal of python2 support > > > > * Removal of hard dependency on HDP > > > > * Working on bigtop mpack > > > > > > > > Can add more based on your inputs… > > > > > > > > > > > > Any thoughts on this..? Please feel free to add anything which I missed. > > > > > > > > > > > > > > > > Reference: > > > > > > > > 1. > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7Ccb99f7f4857f4aa0c30908da52a8c1d6%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913182006020386%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s%2FI1zyAlaiuvWqx%2FjZo%2FNBu4KeHrPVmxtIfeYgLQAso%3D&reserved=0 > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
