And BTW. We are going to have a common talk by the stakeholders who have their dashboards of ours at Airflow Summit https://airflowsummit.org in September, explaining how having such a dashboard managed by the stakeholders improves the quality of provider code. I guess it won't be possible to get such a dashboard from Microsoft before, but at least mentioning that Microsoft is working on a similar solution would be a good point at the talk - it might give the users assurance that the stakeholders are taking care of them.
BTW. Airflow Summit also still looks for sponsors and while Amazon, Google and Astronomer are "organizers" and Platinum sponsors of the Summit, the other stakeholders are more than welcome to become sponsors as well. Would be great to see Microsoft as a sponsor there. J. On Sat, Jul 27, 2024 at 11:35 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Sure. We have everything documented, agreed and approved by voting of the > community here. Generally I strongly advice subscribing to the devlist and > following what's happening here. That is basically a prerequisite of "being > part of the community". > > System tests are described here: > > > https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst > > They are the result of implementing the AIP-4 (Airflow Improvement > Proposal) > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93324355. > - which had recently been completed with Google and Terraform publishing > their dashboards (Amazon and Astronomer implemented them before) > > Everything about providers is described here: > https://github.com/apache/airflow/blob/main/PROVIDERS.rst (+ links to > follow) > > That includes "mixed governance" model I referred to earlier > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#mixed-governance-model-for-3rd-party-related-community-providers > > Also the available dashboards from Amazon, Google, Astronomer and Teradata > are available here > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > - we do not tell our stakeholders how to develop dashboards (because for > each of them the dashboard can be implemented best by following their own > "usual" ways. The expectation we have that the "system tests" (also known > as example dags) are executed with the latest main against the "real" > services and the result of this run is published so that the release > manager releasing the providers can take a look before releasing the > provider - maintaining, reacting to issues and fixing them is done > within the community (by raising PRs and fixes) but should be driven by the > stakeholder. > > There is also a discussion started by Dennis Ferruzzi from Amazon about > exposing machine-readable way of exposing dashboard results so that they > can be aggregated and used as a driver for our CI jobs > https://lists.apache.org/thread/rx59zjvf3m3nkhq44c93hpttjcn1s85s - which > might be acutally something we might seriously consider - especialy that we > got another contribution from Amazon on our AWS CI infrastructure which we > can utilise for this. This is one of the projects that a "newcomer" in this > area (Microsoft I mean) might consider as a project to lead - it would be a > great contribution to the whole community if standard for this reporting is > proposed and shared between the stakeholders and we have one common way of > reporting it - contributions and engineering and project management effort > in this area are most welcome, that could be another way Microsoft could > show that they indeed want to contribute back and engage with the community. > > J. > > > > > > > On Sat, Jul 27, 2024 at 11:04 PM Abhishek Narain < > narain.abhis...@microsoft.com> wrote: > >> Sorry for the lack of context. Could you share documentation around >> setting up dashboards so that we can assess the work required and get back? >> >> Get Outlook for iOS <https://aka.ms/o0ukef> >> ------------------------------ >> *From:* Jarek Potiuk <ja...@potiuk.com> >> *Sent:* Saturday, July 27, 2024 1:48 PM >> *To:* dev@airflow.apache.org <dev@airflow.apache.org>; Abhishek Narain < >> narain.abhis...@microsoft.com> >> *Cc:* ambika garg <ambikagarg1...@gmail.com> >> *Subject:* Re: [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric >> Provider in Apache Airflow >> >> You don't often get email from ja...@potiuk.com. Learn why this is >> important <https://aka.ms/LearnAboutSenderIdentification> >> Adding Abishek explicitly and repeating the message: >> >> Hello Abishek, >> >> Does it mean that you are going to build and expose the dashboard with >> system tests running on Microsoft Azure provider - similarly as Amazon, >> Astronomer, Google, Terraform did? >> >> I guess that request will have to wait for it to happen - as a way to >> actually show that the "intention" meets the "actual work done". >> >> Do you agree it's a great prerequisite before adding a new provider? (and >> BTW - the new provider should also be only included when the dashboard and >> system tests for it is there). >> >> I think it's a pretty reasonable expectation and a very straightforward >> way to "put your money where your mouth is". >> >> What do you think? >> >> J. >> >> On Sat, Jul 27, 2024 at 10:43 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> Hello Abishek, >>> >>> Does it mean that you are going to build and expose the dashboard with >>> system tests running on Microsoft Azure provider - similarly as Amazon, >>> Astronomer, Google, Terraform did? >>> >>> I guess that request will have to wait for it to happen - as a way to >>> actually show that the "intention" meets the "actual work done". >>> >>> Do you agree it's a great prerequisite before adding a new provider? >>> (and BTW - the new provider should also be only included when the dashboard >>> and system tests for it is there). >>> >>> I think it's a pretty reasonable expectation and a very straightforward >>> way to "put your money where your mouth is". >>> >>> What do you think? >>> >>> J. >>> >>> On Sat, Jul 27, 2024 at 2:23 PM Abhishek Narain >>> <narain.abhis...@microsoft.com.invalid> wrote: >>> >>>> Thank you for your detailed response, Ambika. >>>> >>>> I would like to reaffirm Microsoft’s commitment to supporting the >>>> Fabric Provider for Apache Airflow. Our team is already dedicated to >>>> delivering a native Apache Airflow powered SaaS experience known as Data >>>> Workflows in Microsoft Fabric >>>> <https://blog.fabric.microsoft.com/en-us/blog/introducing-data-workflows-in-microsoft-fabric?ft=All>, >>>> which is currently in public preview. By adding a native provider, we will >>>> not only be building this for the community but also using it natively to >>>> better integrate Data Workflows to run other Fabric jobs. >>>> >>>> We are excited to collaborate with the community to ensure the success >>>> and sustainability of the Microsoft Fabric provider. >>>> >>>> >>>> Regards, >>>> >>>> *Abhishek Narain* >>>> Principal Product Manager >>>> Data Integration >>>> narain.abhis...@microsoft.com >>>> >>>> >>>> [image: Microsoft Logo] >>>> >>>> [image: Pz8hmF9cQjDg8MHAaiKxp0eiZJ4AAAAAElFTkSuQmCC] >>>> <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> >>>> >>>> Book time to meet with me >>>> <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> >>>> >>>> >>>> >>>> ------------------------------ >>>> *From:* ambika garg <ambikagarg1...@gmail.com> >>>> *Sent:* Wednesday, July 24, 2024 8:21 PM >>>> *To:* dev@airflow.apache.org <dev@airflow.apache.org>; Abhishek Narain >>>> <narain.abhis...@microsoft.com> >>>> *Subject:* [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric Provider >>>> in Apache Airflow >>>> >>>> Thank you for your thoughtful and detailed inquiry regarding the >>>> maintainability and support for the proposed Microsoft Fabric provider for >>>> Apache Airflow. I appreciate your commitment to transparency and >>>> collaboration within the Apache Airflow community. Your concerns about >>>> long-term support and the active involvement of stakeholders are entirely >>>> valid. >>>> >>>> To address your questions directly: >>>> >>>> 1. *Affiliation with Microsoft:* I am working as a vendor at >>>> Microsoft, and I am actively contributing to this proposal on behalf of the >>>> company. I have also developed the private plugin for Microsoft Fabric >>>> <https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin>, >>>> that triggers run of on-demand fabric items. Previously, I have worked on >>>> Synapse >>>> operator that triggers pipeline run >>>> <https://github.com/apache/airflow/pull/35091> and my new PR that >>>> integrates the Power BI operator for dataset refresh >>>> <https://github.com/apache/airflow/pull/40356> is in-progress. >>>> However, I understand that the community's interest is not just in >>>> individual contributors but in the sustained involvement and support from >>>> the organization. >>>> >>>> 2. *Microsoft’s Commitment:* I can assure you that this proposal is >>>> backed by a commitment from Microsoft to not only contribute the initial >>>> code but also to engage actively in its long-term maintenance and >>>> integration. >>>> >>>> To ensure the community has full visibility into Microsoft’s commitment >>>> and future plans, I have reached out to my manager. I am tagging >>>> @abnar...@microsoft.com <abnar...@microsoft.com> Abhishek Narain, who >>>> can provide more detailed information about our roadmap, resources, and >>>> long-term strategy for supporting this integration. >>>> >>>> Thank you again for your patience. We are eager to work alongside the >>>> community to ensure the success and sustainability of the Microsoft Fabric >>>> provider. >>>> >>>> Best regards, >>>> Ambika Garg >>>> >>>
