Here’s where I’m at . https://github.com/jayunit100/apachecon-2019-bigtop3x/
It’s a lot easier then packaging all this stuff, but for most people will accomplish a similar end goal ( a big Data cloud in a box) and mostly a matter of curation of charts and images, and integration testing . So I see the amount of code to maintain for bigtop becoming minimal, and most of our job being curation and integration testing of canonical releases of these images and helm charts / operator yamls. This will be a significant paradigm shift if we do it , but it will be something that we can maintain even if we have a small community , that adds differntiated value to the asf and cncf ecosystems for sure. Working with the minio folks on how presto integration will work. Come to apachecon Vegas and we can talk more about a plan for the future . > On Aug 29, 2019, at 9:28 PM, Jun HE <[email protected]> wrote: > > Super +1 for Bigtop 2.0 with modern components and K8S support! I cannot > wait to see Bigtop running on K8S... > So can someone help to create a branch-2.0 now? :P > > > Konstantin Boudnik <[email protected]> 于2019年8月30日周五 上午2:24写道: > >> Aah, k8s - thanks Jay! >> >> BTW, I believe we can do this in a less formal stuff (ie no VOTE): we have >> a >> discussion going on the version of Bigtop. How about we make it 2.0 and >> trim >> the BOM into the needed shape and form? If there's enough drive in the >> community to continue with 1.x line (1.4 and so on), it can be done as a >> parallel effort. >> >> Thoughts? >> Cos >> >>> On Fri, Aug 30, 2019 at 01:26AM, Evans Ye wrote: >>> I'm super +1 with K8S stuff and the core components idea! >>> >>> I'm with Cos and Jun. Removing those stuffs can ease our CI resource >>> utilization and yield more stable CI results. This is what I'm thinking >> how >>> to proceed based on the previous feedback that a drop of supporting >>> component should be carefully discussed >>> >>> 1. Spin up a vote to drop project XXX (if confident enough, put multiple >>> components here) >>> 2. If +3 binding votes with no -1 votes, it passes. >>> 3. Put up the PR for code removing >>> 4. Refine our CI settings (I can definitely help with this part) >>> >>> Best, >>> Evans >>> >>> >>> >>> >>> Jay Vyas <[email protected]> 於 2019年8月29日 週四 下午7:26寫道: >>> >>>> First of all I’m all for dropping the old stuff. >>>> >>>> 1) My opinion - to further cos point - I think people running old stuff >>>> don’t really need version updates, or if they do, they don’t >> constitute a >>>> large audience , or should contribute them ok their own. >>>> >>>> 2) surprise surprise :):) here’s my k8s native. View - we should move >> the >>>> old components into sustaining mode, and rebuild a new bigtop focus >> solely >>>> on spark , NiFi , Presto (with a lot of attention to minimal standalone >>>> Hadoop w/ a Hive connector ) and target it at kubernetes native >> deployments >>>> :). >>>> >>>> I can make it so the k8s part is an impl detail so users can be mostly >>>> decoupled from it . >>>> >>>> I’ve been integration testing various things in the bigdata ecosystem >> on >>>> k8s and there is a lot of demand without anyone owning the integration. >>>> >>>>> On Aug 29, 2019, at 2:39 AM, Konstantin Boudnik <[email protected]> >> wrote: >>>>> >>>>> I am not sure about Giraph, Tajo and some others, but Sqoop seems to >> be >>>> user >>>>> around by people. So, if it isn't much of the burden for us - and it >>>> seems >>>>> pretty stable at the moment - I'd leave it. >>>>> >>>>> What I would think would makes sense to spend some of our efforts on >> is >>>> on >>>>> adding modern tooling like Nifi/Airflow into the mix. As you said - >>>> things >>>>> move forward pretty fast, and we seem to be sticking to the some of >> the >>>> old >>>>> stuff. >>>>> >>>>> Thanks for starting this discussion! >>>>> Cos >>>>> >>>>>> On Wed, Aug 28, 2019 at 04:12PM, Jun HE wrote: >>>>>> Hi, folks, >>>>>> >>>>>> I went through current components Bigtop is supporting, and I >> noticed >>>> that >>>>>> these upstream projects haven't been released for quite a while: >>>>>> Apache Tajo: >>>>>> last release: release-0.11.3-rc0, May 11, 2016 >>>>>> Apache Apex: >>>>>> last release: v3.7.0, Apr 27, 2018 >>>>>> Apache Giraph: >>>>>> last release: rel/1.2.0-RC1, Oct 13 2016 >>>>>> Apache Hama: >>>>>> last release: 0.7.1-RC2, Mar 12, 2016 >>>>>> Apache Sqoop: >>>>>> last release: release-1.4.7-rc0, Dec 6, 2017; >> release-1.99.7-rc1, Jul >>>>>> 20, 2016 >>>>>> >>>>>> And some of them seem to be in slow development: >>>>>> Apache Tajo: >>>>>> last commit: Jul 13, 2018 >>>>>> Apache Apex: >>>>>> last commit: Jun 20, 2018 >>>>>> Apache Hama: >>>>>> last commit: Jul 30, 2018 >>>>>> >>>>>> So I'm wondering whether we should continue support for these >> components >>>>>> (or part of them) in next/future releases. >>>>>> >>>>>> I understand that similar topics were discussed before. But, you >> know, >>>> this >>>>>> is an quickly evolving world, maybe it worth another revisit now? ;) >>>>>> >>>>>> Regards, >>>>>> >>>>>> Jun >>>> >>
