Hi, Looks like a good idea but before committing myself, I have a number of design questions having looked at SPIP itself:
1. Will the name "Standard add-on Kubernetes operator to Spark '' describe it better? 2. We are still struggling with improving Spark driver start-up time. What would be the footprint of this add-on on the driver start-up time? 3. In a commercial world will there be (?) a static image for this besides the base image that is maintained in the so called container registry (ECR, GCR etc), It takes time to upload these images. Will this bea static image (docker file)? Other alternative would be that this docker file is created by the user through set of scripts? These are the things that come into my mind. HTH Mich Talebzadeh, Distinguished Technologist, Solutions Architect & Engineer London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Nov 2023 at 14:19, Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > +1 > > fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu <zhunanmcg...@gmail.com>: > >> just curious what happened on google’s spark operator? >> >> On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko <i...@cornell.edu> wrote: >> >>> +1 >>> >>> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue <b...@tabular.io> wrote: >>> >>>> +1 >>>> >>>> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala <huss...@awala.fr> wrote: >>>> >>>>> +1 for creating an official Kubernetes operator for Apache Spark >>>>> >>>>> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao <huaxin.ga...@gmail.com> >>>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>> >>>>>> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai <dbt...@dbtsai.com> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> To be completely transparent, I am employed in the same department >>>>>>> as Zhou at Apple. >>>>>>> >>>>>>> I support this proposal, provided that we witness community adoption >>>>>>> following the release of the Flink Kubernetes operator, streamlining >>>>>>> Flink >>>>>>> deployment on Kubernetes. >>>>>>> >>>>>>> A well-maintained official Spark Kubernetes operator is essential >>>>>>> for our Spark community as well. >>>>>>> >>>>>>> DB Tsai | https://www.dbtsai.com/ >>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dbtsai.com%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uZSpzGB3TcMkiB4aGlteedWlk%2FL3M8XgHfcFxasEGUk%3D&reserved=0> >>>>>>> | PGP 42E5B25A8F7A82C1 >>>>>>> >>>>>>> On Nov 9, 2023, at 12:05 PM, Zhou Jiang <zhou.c.ji...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Spark community, >>>>>>> I'm reaching out to initiate a conversation about the possibility of >>>>>>> developing a Java-based Kubernetes operator for Apache Spark. Following >>>>>>> the >>>>>>> operator pattern ( >>>>>>> https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ >>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkubernetes.io%2Fdocs%2Fconcepts%2Fextend-kubernetes%2Foperator%2F&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Np4pJPeJNqKLEJWsH5PrGQ%2FxbcbQXs6lk8i5pCgMkaE%3D&reserved=0>), >>>>>>> Spark users may manage applications and related components seamlessly >>>>>>> using >>>>>>> native tools like kubectl. The primary goal is to simplify the Spark >>>>>>> user >>>>>>> experience on Kubernetes, minimizing the learning curve and operational >>>>>>> complexities and therefore enable users to focus on the Spark >>>>>>> application >>>>>>> development. >>>>>>> Although there are several open-source Spark on Kubernetes operators >>>>>>> available, none of them are officially integrated into the Apache Spark >>>>>>> project. As a result, these operators may lack active support and >>>>>>> development for new features. Within this proposal, our aim is to >>>>>>> introduce >>>>>>> a Java-based Spark operator as an integral component of the Apache Spark >>>>>>> project. This solution has been employed internally at Apple for >>>>>>> multiple >>>>>>> years, operating millions of executors in real production environments. >>>>>>> The >>>>>>> use of Java in this solution is intended to accommodate a wider user and >>>>>>> contributor audience, especially those who are familiar with Scala. >>>>>>> Ideally, this operator should have its dedicated repository, similar >>>>>>> to Spark Connect Golang or Spark Docker, allowing it to maintain a loose >>>>>>> connection with the Spark release cycle. This model is also followed by >>>>>>> the >>>>>>> Apache Flink Kubernetes operator. >>>>>>> We believe that this project holds the potential to evolve into a >>>>>>> thriving community project over the long run. A comparison can be drawn >>>>>>> with the Flink Kubernetes Operator: Apple has open-sourced internal >>>>>>> Flink >>>>>>> Kubernetes operator, making it a part of the Apache Flink project ( >>>>>>> https://github.com/apache/flink-kubernetes-operator >>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-kubernetes-operator&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jltCb10Ws2CxEHh4%2FF%2Big96Tt8U1UCEZlmhAuWRxx9Y%3D&reserved=0>). >>>>>>> This move has gained wide industry adoption and contributions from the >>>>>>> community. In a mere year, the Flink operator has garnered more than 600 >>>>>>> stars and has attracted contributions from over 80 contributors. This >>>>>>> showcases the level of community interest and collaborative momentum >>>>>>> that >>>>>>> can be achieved in similar scenarios. >>>>>>> More details can be found at SPIP doc : Spark Kubernetes Operator >>>>>>> https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE >>>>>>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE&data=05%7C01%7Cif56%40g.cornell.edu%7C6b33babc19c64437ef0408dbe18607c6%7C5d7e43661b9b45cf8e79b14b27df46e1%7C0%7C0%7C638351737993352064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=w8FrIp88nEpI7lXCBy7Y2U9NZ0uy%2B2Bssu7wjFqZCFw%3D&reserved=0> >>>>>>> >>>>>>> Thanks, >>>>>>> -- >>>>>>> *Zhou JIANG* >>>>>>> >>>>>>> >>>>>>> >>>> >>>> -- >>>> Ryan Blue >>>> Tabular >>>> >>> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 >
