+1 Kazu
> On Nov 10, 2023, at 10:05 AM, Khalid Mammadov <khalidmammad...@gmail.com> > wrote: > > +1 > > On Fri, 10 Nov 2023, 15:23 Peter Toth, <peter.t...@gmail.com > <mailto:peter.t...@gmail.com>> wrote: >> +1 >> >> On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen <bjornjorgen...@gmail.com >> <mailto:bjornjorgen...@gmail.com>> wrote: >>> +1 >>> >>> fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu <zhunanmcg...@gmail.com >>> <mailto:zhunanmcg...@gmail.com>>: >>>> just curious what happened on google’s spark operator? >>>> >>>> On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko <i...@cornell.edu >>>> <mailto:i...@cornell.edu>> wrote: >>>>> +1 >>>>> >>>>> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue <b...@tabular.io >>>>> <mailto:b...@tabular.io>> wrote: >>>>>> +1 >>>>>> >>>>>> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala <huss...@awala.fr >>>>>> <mailto:huss...@awala.fr>> wrote: >>>>>>> +1 for creating an official Kubernetes operator for Apache Spark >>>>>>> >>>>>>> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao <huaxin.ga...@gmail.com >>>>>>> <mailto:huaxin.ga...@gmail.com>> wrote: >>>>>>>> +1 >>>>> >>>>>>>> >>>>>>>> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai <dbt...@dbtsai.com >>>>>>>> <mailto:dbt...@dbtsai.com>> wrote: >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> To be completely transparent, I am employed in the same department as >>>>>>>>> Zhou at Apple. >>>>>>>>> >>>>>>>>> I support this proposal, provided that we witness community adoption >>>>>>>>> following the release of the Flink Kubernetes operator, streamlining >>>>>>>>> Flink deployment on Kubernetes. >>>>>>>>> >>>>>>>>> A well-maintained official Spark Kubernetes operator is essential for >>>>>>>>> our Spark community as well. >>>>>>>>> >>>>>>>>> DB Tsai | https://www.dbtsai.com/ >>>>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dbtsai.com%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uZSpzGB3TcMkiB4aGlteedWlk%2FL3M8XgHfcFxasEGUk%3D&reserved=0> >>>>>>>>> | PGP 42E5B25A8F7A82C1 >>>>>>>>> >>>>>>>>>> On Nov 9, 2023, at 12:05 PM, Zhou Jiang <zhou.c.ji...@gmail.com >>>>>>>>>> <mailto:zhou.c.ji...@gmail.com>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Spark community, >>>>>>>>>> I'm reaching out to initiate a conversation about the possibility of >>>>>>>>>> developing a Java-based Kubernetes operator for Apache Spark. >>>>>>>>>> Following the operator pattern >>>>>>>>>> (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ >>>>>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkubernetes.io%2Fdocs%2Fconcepts%2Fextend-kubernetes%2Foperator%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Np4pJPeJNqKLEJWsH5PrGQ%2FxbcbQXs6lk8i5pCgMkaE%3D&reserved=0>), >>>>>>>>>> Spark users may manage applications and related components >>>>>>>>>> seamlessly using native tools like kubectl. The primary goal is to >>>>>>>>>> simplify the Spark user experience on Kubernetes, minimizing the >>>>>>>>>> learning curve and operational complexities and therefore enable >>>>>>>>>> users to focus on the Spark application development. >>>>>>>>>> Although there are several open-source Spark on Kubernetes operators >>>>>>>>>> available, none of them are officially integrated into the Apache >>>>>>>>>> Spark project. As a result, these operators may lack active support >>>>>>>>>> and development for new features. Within this proposal, our aim is >>>>>>>>>> to introduce a Java-based Spark operator as an integral component of >>>>>>>>>> the Apache Spark project. This solution has been employed internally >>>>>>>>>> at Apple for multiple years, operating millions of executors in real >>>>>>>>>> production environments. The use of Java in this solution is >>>>>>>>>> intended to accommodate a wider user and contributor audience, >>>>>>>>>> especially those who are familiar with Scala. >>>>>>>>>> Ideally, this operator should have its dedicated repository, similar >>>>>>>>>> to Spark Connect Golang or Spark Docker, allowing it to maintain a >>>>>>>>>> loose connection with the Spark release cycle. This model is also >>>>>>>>>> followed by the Apache Flink Kubernetes operator. >>>>>>>>>> We believe that this project holds the potential to evolve into a >>>>>>>>>> thriving community project over the long run. A comparison can be >>>>>>>>>> drawn with the Flink Kubernetes Operator: Apple has open-sourced >>>>>>>>>> internal Flink Kubernetes operator, making it a part of the Apache >>>>>>>>>> Flink project (https://github.com/apache/flink-kubernetes-operator >>>>>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-kubernetes-operator&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jltCb10Ws2CxEHh4%2FF%2Big96Tt8U1UCEZlmhAuWRxx9Y%3D&reserved=0>). >>>>>>>>>> This move has gained wide industry adoption and contributions from >>>>>>>>>> the community. In a mere year, the Flink operator has garnered more >>>>>>>>>> than 600 stars and has attracted contributions from over 80 >>>>>>>>>> contributors. This showcases the level of community interest and >>>>>>>>>> collaborative momentum that can be achieved in similar scenarios. >>>>>>>>>> More details can be found at SPIP doc : Spark Kubernetes Operator >>>>>>>>>> https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE >>>>>>>>>> >>>>>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w8FrIp88nEpI7lXCBy7Y2U9NZ0uy%2B2Bssu7wjFqZCFw%3D&reserved=0> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Zhou JIANG >>>>>>>>>> >>>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>> >>> >>> -- >>> Bjørn Jørgensen >>> Vestre Aspehaug 4, 6010 Ålesund >>> Norge >>> >>> +47 480 94 297
