Jarek and XD, Thank you for your responses. I have read through the other thread, and understand the concerns with accepting a submission to the Airflow community providers. I believe we can address these concerns, as we see value in being included within the infrastructure. As I mentioned before, we plan to fully test, maintain and support our provider, as it is in our best interests to do so. SAS has a number of customers across the globe that extensively use Airflow in their organizations and want to expand Airflow usage to orchestrate and schedule SAS workloads. We want to make it as easy as possible for them, and we want our Airflow provider to be as discoverable as possible.
As far as community testing is concerned, SAS provides trial offerings: https://www.sas.com/en_us/trials.html, so operator can be tested by anyone. SAS Studio is an Integrated Development Environment for Data Engineers and Data Scientists on SAS Viya. SAS Studio Flow is a component and an artifact of SAS Studio used for building visual data pipelines in a low code environment. The Studio Flow Airflow operator that we have developed allows the execution SAS Studio Flows within SAS Viya infrastructure and provides the ability to specify parameters needed for execution. Our operator is generic and not tied to a specific Studio Flow. Further, our operator uses only public documented SAS Viya APIs available through the SAS Developer Portal: https://developer.sas.com/apis/rest/ Please let me know if there are additional concerns. I will submit a PR as well so you can take a look at what we have. Thanks From: Jarek Potiuk <ja...@potiuk.com> Date: Friday, December 16, 2022 at 1:05 PM To: dev@airflow.apache.org <dev@airflow.apache.org> Cc: andrew.shakinov...@sas.com.invalid <andrew.shakinov...@sas.com.invalid> Subject: Re: New provider for SAS You don't often get email from ja...@potiuk.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> EXTERNAL Yep. I think the important one is that I believe (or at least this is my proposal to address some of the concerns of the maintainers) we will be leaning towards the approach that if we accept any submission for an "external service" provider, then there must be a commitment from the service provider to create and maintain "System test harness" on their own. And the proposal is that they should implement "System Tests" in the provider that will allow us in the community to know that their provider is still functioning with the external service. That's pretty high bar, I think, and some of our service providers - Google, Amazon, Databricks - are already committed to do that for their providers (and even they took part in making the System tests harness robust and usable in such scenarios). Staying with your "own" way of relassing and maintaining such a provider is far easier and bears no such long time commitments, you have more freedom and more control, while this means that it can only be listed in the "ecosystem" part of the official Airflow documentation. Note - this is not yet a formalized and agreed policy, but I personally think it is one that has the right balance of service providers needs and community expectations. J. On Fri, Dec 16, 2022 at 4:38 PM Xiaodong Deng <xdd...@apache.org<mailto:xdd...@apache.org>> wrote: Hi Andrew, Thanks a lot for the email and your interest to make the contribution. Earlier we did have a similar conversation for another potential provider, at https://lists.apache.org/thread/1gtw5vyypxh0p72wh4dss7cllcvhgh01<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2F1gtw5vyypxh0p72wh4dss7cllcvhgh01&data=05%7C01%7CAndrew.Shakinovsky%40sas.com%7C08491a4359a04847a96308dadf9022fc%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638068107422828095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UNmIuIvB48QMWUTUORpq3srJ0g6UIwunrbWUBENctFg%3D&reserved=0> . Inside I believe you will find a lot of useful information. In short, we would like to be more careful when we review such contributions, because accepting it also means long-term commitment from both the Airflow maintainer side & the provider contributor side. In the email exchange I shared above, you will find more detailed information about our concern (but of course none of them is an immediate blocker). Hope it helps. Regards, XD On Fri, Dec 16, 2022 at 5:43 AM Andrew Shakinovsky <andrew.shakinov...@sas.com.invalid> wrote: Hi all, We have developed a provider to allow execution of SAS jobs and flows from Airflow. This provider will be primarily useful for our customers to integrate with Airflow. We plan to maintain and support it moving forward. Having read through some contributing doc, I feel it would be appropriate to have it hosted in with the Airflow community providers in the main tree. I am planning to submit a PR for this, but wanted to make sure I am on the right track. Is there anything else I should consider or know? Thanks
