>*>1. Deployment on K8S: Ambari or Bigtop Puppet as K8S operators.* Agreed with Youngwoo that the k8s operator would be a good candidate to automate the management of the entire lifecycle of our stack. And just as Luca mentioned, it's not easy to write a new operator. IMO, we'd better firstly check if there is already a good operator out there doing the job. (https://operatorhub.io/). If not, we may make use of k8s operator sdk <https://github.com/operator-framework/operator-sdk> ( or operator SDK based on Java <https://github.com/java-operator-sdk/java-operator-sdk>) to implement our own k8s operator. It is challenging but interesting work for us. :)
BRs, Yuqi Masatake Iwasaki <[email protected]> 于2021年11月9日周二 下午10:07写道： > > Hi Kengo/Masatake, > > Do you have any feature that is needed for your company to move business > > forward? > > Maintaining "the open source ver. of unified compatible Hadoop Distro" for > a long-term > and more frequent release is my primary concern. > Cleaning up obsolete features/products would help that. > Adding Ozone (BIGTOP-3445) could be added value for ML workload too. > > Thanks, > Masatake Iwasaki > > On 2021/11/05 22:53, Evans Ye wrote: > > Hi Matt Andruff, > > Agree. > > > > Hi Youngwoo, > > Could you elaborate what component is needed for the real-time event > > streaming? Kafka + Flink in current stack are the solution for it. Pulsar > > can be an addition. > > Regarding query processing, Could you share more insight on the > difference > > of Pinot V.S. Presto? Does Pinot suitable for having Looker plugged in > > front of it for analytical purposes? > > > > Hi Kengo/Masatake, > > Do you have any feature that is needed for your company to move business > > forward? > > > > > > BTW, we have some discussion in the past for this topic you can take as a > > reference[1]. > > > > [1] > > > https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit# > > > > Youngwoo Kim (김영우) <[email protected]> 於 2021年11月3日 週三 下午4:02寫道： > > > >> Evans, > >> Thanks for starting this discussion. > >> > >> Hopefully, It would be valuable to integrate the real-time event > streaming > >> and query processing stack e.g., Apache Druid, Pinot, Pulsar and etc. > >> > >> And 'k8s operator for Bigtop' looks promising for me! > >> > >> Thanks, > >> Youngwoo > >> > >> On Wed, Nov 3, 2021 at 1:34 AM Evans Ye <[email protected]> wrote: > >> > >>> Hi folks, > >>> > >>> With Bigtop 3.0 been released, I think it's time to discuss what's new > as > >>> our next steps. Of course the open source ver. of unified compatible > >> Hadoop > >>> Distro. is still our core product going forward. But the surrounding > >> value > >>> added features might be something that can take us further beyond where > >> we > >>> were at. Now, let me post some ideas to start the brainstorming. > >>> > >>> 1. Deployment on K8S: Ambari or Bigtop Puppet as K8S operators. > >>> 2. MLOps integrations: MLFlow, Submarine. > >>> 3. Data Lake integrations: Hudi, Iceberg, Delta. > >>> > >>> And for some software engineering stuffs, I think we can do a clean up > on > >>> out-dated features such as: > >>> 1. vagrant provisioner > >>> 2. docker sandbox > >>> 3. bigtop-ci > >>> 4. bigtop-data-generators > >>> 5. bigtop-bigpetstore > >>> > >>> Any thoughts? Would love to hear all of you. > >>> > >> > > >
