+1 On Fri, Nov 10, 2023 at 10:01 AM Ilan Filonenko <i...@cornell.edu> wrote:
> +1 > > On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue <b...@tabular.io> wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala <huss...@awala.fr> wrote: >> >>> +1 for creating an official Kubernetes operator for Apache Spark >>> >>> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao <huaxin.ga...@gmail.com> >>> wrote: >>> >>>> +1 >>>> >>>> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai <dbt...@dbtsai.com> wrote: >>>> >>>>> +1 >>>>> >>>>> To be completely transparent, I am employed in the same department as >>>>> Zhou at Apple. >>>>> >>>>> I support this proposal, provided that we witness community adoption >>>>> following the release of the Flink Kubernetes operator, streamlining Flink >>>>> deployment on Kubernetes. >>>>> >>>>> A well-maintained official Spark Kubernetes operator is essential for >>>>> our Spark community as well. >>>>> >>>>> DB Tsai | https://www.dbtsai.com/ >>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dbtsai.com%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uZSpzGB3TcMkiB4aGlteedWlk%2FL3M8XgHfcFxasEGUk%3D&reserved=0> >>>>> | PGP 42E5B25A8F7A82C1 >>>>> >>>>> On Nov 9, 2023, at 12:05 PM, Zhou Jiang <zhou.c.ji...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi Spark community, >>>>> I'm reaching out to initiate a conversation about the possibility of >>>>> developing a Java-based Kubernetes operator for Apache Spark. Following >>>>> the >>>>> operator pattern ( >>>>> https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ >>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkubernetes.io%2Fdocs%2Fconcepts%2Fextend-kubernetes%2Foperator%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Np4pJPeJNqKLEJWsH5PrGQ%2FxbcbQXs6lk8i5pCgMkaE%3D&reserved=0>), >>>>> Spark users may manage applications and related components seamlessly >>>>> using >>>>> native tools like kubectl. The primary goal is to simplify the Spark user >>>>> experience on Kubernetes, minimizing the learning curve and operational >>>>> complexities and therefore enable users to focus on the Spark application >>>>> development. >>>>> Although there are several open-source Spark on Kubernetes operators >>>>> available, none of them are officially integrated into the Apache Spark >>>>> project. As a result, these operators may lack active support and >>>>> development for new features. Within this proposal, our aim is to >>>>> introduce >>>>> a Java-based Spark operator as an integral component of the Apache Spark >>>>> project. This solution has been employed internally at Apple for multiple >>>>> years, operating millions of executors in real production environments. >>>>> The >>>>> use of Java in this solution is intended to accommodate a wider user and >>>>> contributor audience, especially those who are familiar with Scala. >>>>> Ideally, this operator should have its dedicated repository, similar >>>>> to Spark Connect Golang or Spark Docker, allowing it to maintain a loose >>>>> connection with the Spark release cycle. This model is also followed by >>>>> the >>>>> Apache Flink Kubernetes operator. >>>>> We believe that this project holds the potential to evolve into a >>>>> thriving community project over the long run. A comparison can be drawn >>>>> with the Flink Kubernetes Operator: Apple has open-sourced internal Flink >>>>> Kubernetes operator, making it a part of the Apache Flink project ( >>>>> https://github.com/apache/flink-kubernetes-operator >>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-kubernetes-operator&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jltCb10Ws2CxEHh4%2FF%2Big96Tt8U1UCEZlmhAuWRxx9Y%3D&reserved=0>). >>>>> This move has gained wide industry adoption and contributions from the >>>>> community. In a mere year, the Flink operator has garnered more than 600 >>>>> stars and has attracted contributions from over 80 contributors. This >>>>> showcases the level of community interest and collaborative momentum that >>>>> can be achieved in similar scenarios. >>>>> More details can be found at SPIP doc : Spark Kubernetes Operator >>>>> https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE >>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w8FrIp88nEpI7lXCBy7Y2U9NZ0uy%2B2Bssu7wjFqZCFw%3D&reserved=0> >>>>> >>>>> Thanks, >>>>> -- >>>>> *Zhou JIANG* >>>>> >>>>> >>>>> >> >> -- >> Ryan Blue >> Tabular >> >
