So - plenty of ways to show that "you are serious about contributing back". The ball is on your court now.
On Sat, Jul 27, 2024 at 11:43 PM Jarek Potiuk <ja...@potiuk.com> wrote: > And BTW. We are going to have a common talk by the stakeholders who have > their dashboards of ours at Airflow Summit https://airflowsummit.org in > September, explaining how having such a dashboard managed by the > stakeholders improves the quality of provider code. I guess it won't be > possible to get such a dashboard from Microsoft before, but at least > mentioning that Microsoft is working on a similar solution would be a good > point at the talk - it might give the users assurance that the stakeholders > are taking care of them. > > BTW. Airflow Summit also still looks for sponsors and while Amazon, Google > and Astronomer are "organizers" and Platinum sponsors of the Summit, the > other stakeholders are more than welcome to become sponsors as well. Would > be great to see Microsoft as a sponsor there. > > J. > > On Sat, Jul 27, 2024 at 11:35 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Sure. We have everything documented, agreed and approved by voting of the >> community here. Generally I strongly advice subscribing to the devlist and >> following what's happening here. That is basically a prerequisite of "being >> part of the community". >> >> System tests are described here: >> >> >> https://github.com/apache/airflow/blob/main/contributing-docs/testing/system_tests.rst >> >> They are the result of implementing the AIP-4 (Airflow Improvement >> Proposal) >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=93324355. >> - which had recently been completed with Google and Terraform publishing >> their dashboards (Amazon and Astronomer implemented them before) >> >> Everything about providers is described here: >> https://github.com/apache/airflow/blob/main/PROVIDERS.rst (+ links to >> follow) >> >> That includes "mixed governance" model I referred to earlier >> https://github.com/apache/airflow/blob/main/PROVIDERS.rst#mixed-governance-model-for-3rd-party-related-community-providers >> >> Also the available dashboards from Amazon, Google, Astronomer and >> Teradata are available here >> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards >> - we do not tell our stakeholders how to develop dashboards (because for >> each of them the dashboard can be implemented best by following their own >> "usual" ways. The expectation we have that the "system tests" (also known >> as example dags) are executed with the latest main against the "real" >> services and the result of this run is published so that the release >> manager releasing the providers can take a look before releasing the >> provider - maintaining, reacting to issues and fixing them is done >> within the community (by raising PRs and fixes) but should be driven by the >> stakeholder. >> >> There is also a discussion started by Dennis Ferruzzi from Amazon about >> exposing machine-readable way of exposing dashboard results so that they >> can be aggregated and used as a driver for our CI jobs >> https://lists.apache.org/thread/rx59zjvf3m3nkhq44c93hpttjcn1s85s - >> which might be acutally something we might seriously consider - especialy >> that we got another contribution from Amazon on our AWS CI infrastructure >> which we can utilise for this. This is one of the projects that a >> "newcomer" in this area (Microsoft I mean) might consider as a project to >> lead - it would be a great contribution to the whole community if standard >> for this reporting is proposed and shared between the stakeholders and we >> have one common way of reporting it - contributions and engineering and >> project management effort in this area are most welcome, that could be >> another way Microsoft could show that they indeed want to contribute back >> and engage with the community. >> >> J. >> >> >> >> >> >> >> On Sat, Jul 27, 2024 at 11:04 PM Abhishek Narain < >> narain.abhis...@microsoft.com> wrote: >> >>> Sorry for the lack of context. Could you share documentation around >>> setting up dashboards so that we can assess the work required and get back? >>> >>> Get Outlook for iOS <https://aka.ms/o0ukef> >>> ------------------------------ >>> *From:* Jarek Potiuk <ja...@potiuk.com> >>> *Sent:* Saturday, July 27, 2024 1:48 PM >>> *To:* dev@airflow.apache.org <dev@airflow.apache.org>; Abhishek Narain < >>> narain.abhis...@microsoft.com> >>> *Cc:* ambika garg <ambikagarg1...@gmail.com> >>> *Subject:* Re: [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric >>> Provider in Apache Airflow >>> >>> You don't often get email from ja...@potiuk.com. Learn why this is >>> important <https://aka.ms/LearnAboutSenderIdentification> >>> Adding Abishek explicitly and repeating the message: >>> >>> Hello Abishek, >>> >>> Does it mean that you are going to build and expose the dashboard with >>> system tests running on Microsoft Azure provider - similarly as Amazon, >>> Astronomer, Google, Terraform did? >>> >>> I guess that request will have to wait for it to happen - as a way to >>> actually show that the "intention" meets the "actual work done". >>> >>> Do you agree it's a great prerequisite before adding a new provider? >>> (and BTW - the new provider should also be only included when the dashboard >>> and system tests for it is there). >>> >>> I think it's a pretty reasonable expectation and a very straightforward >>> way to "put your money where your mouth is". >>> >>> What do you think? >>> >>> J. >>> >>> On Sat, Jul 27, 2024 at 10:43 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> Hello Abishek, >>>> >>>> Does it mean that you are going to build and expose the dashboard with >>>> system tests running on Microsoft Azure provider - similarly as Amazon, >>>> Astronomer, Google, Terraform did? >>>> >>>> I guess that request will have to wait for it to happen - as a way to >>>> actually show that the "intention" meets the "actual work done". >>>> >>>> Do you agree it's a great prerequisite before adding a new provider? >>>> (and BTW - the new provider should also be only included when the dashboard >>>> and system tests for it is there). >>>> >>>> I think it's a pretty reasonable expectation and a very straightforward >>>> way to "put your money where your mouth is". >>>> >>>> What do you think? >>>> >>>> J. >>>> >>>> On Sat, Jul 27, 2024 at 2:23 PM Abhishek Narain >>>> <narain.abhis...@microsoft.com.invalid> wrote: >>>> >>>>> Thank you for your detailed response, Ambika. >>>>> >>>>> I would like to reaffirm Microsoft’s commitment to supporting the >>>>> Fabric Provider for Apache Airflow. Our team is already dedicated to >>>>> delivering a native Apache Airflow powered SaaS experience known as Data >>>>> Workflows in Microsoft Fabric >>>>> <https://blog.fabric.microsoft.com/en-us/blog/introducing-data-workflows-in-microsoft-fabric?ft=All>, >>>>> which is currently in public preview. By adding a native provider, we will >>>>> not only be building this for the community but also using it natively to >>>>> better integrate Data Workflows to run other Fabric jobs. >>>>> >>>>> We are excited to collaborate with the community to ensure the success >>>>> and sustainability of the Microsoft Fabric provider. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> *Abhishek Narain* >>>>> Principal Product Manager >>>>> Data Integration >>>>> narain.abhis...@microsoft.com >>>>> >>>>> >>>>> [image: Microsoft Logo] >>>>> >>>>> [image: Pz8hmF9cQjDg8MHAaiKxp0eiZJ4AAAAAElFTkSuQmCC] >>>>> <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> >>>>> >>>>> Book time to meet with me >>>>> <https://outlook.office.com/bookwithme/user/1e04a093bb8944209a5749e9b82a9...@microsoft.com?anonymous&ep=signature> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* ambika garg <ambikagarg1...@gmail.com> >>>>> *Sent:* Wednesday, July 24, 2024 8:21 PM >>>>> *To:* dev@airflow.apache.org <dev@airflow.apache.org>; Abhishek >>>>> Narain <narain.abhis...@microsoft.com> >>>>> *Subject:* [EXTERNAL] Re: [DISCUSS] Add the Microsoft Fabric Provider >>>>> in Apache Airflow >>>>> >>>>> Thank you for your thoughtful and detailed inquiry regarding the >>>>> maintainability and support for the proposed Microsoft Fabric provider for >>>>> Apache Airflow. I appreciate your commitment to transparency and >>>>> collaboration within the Apache Airflow community. Your concerns about >>>>> long-term support and the active involvement of stakeholders are entirely >>>>> valid. >>>>> >>>>> To address your questions directly: >>>>> >>>>> 1. *Affiliation with Microsoft:* I am working as a vendor at >>>>> Microsoft, and I am actively contributing to this proposal on behalf of >>>>> the >>>>> company. I have also developed the private plugin for Microsoft Fabric >>>>> <https://github.com/ambika-garg/apache-airflow-microsoft-fabric-plugin>, >>>>> that triggers run of on-demand fabric items. Previously, I have worked on >>>>> Synapse >>>>> operator that triggers pipeline run >>>>> <https://github.com/apache/airflow/pull/35091> and my new PR that >>>>> integrates the Power BI operator for dataset refresh >>>>> <https://github.com/apache/airflow/pull/40356> is in-progress. >>>>> However, I understand that the community's interest is not just in >>>>> individual contributors but in the sustained involvement and support from >>>>> the organization. >>>>> >>>>> 2. *Microsoft’s Commitment:* I can assure you that this proposal is >>>>> backed by a commitment from Microsoft to not only contribute the initial >>>>> code but also to engage actively in its long-term maintenance and >>>>> integration. >>>>> >>>>> To ensure the community has full visibility into Microsoft’s >>>>> commitment and future plans, I have reached out to my manager. I am >>>>> tagging >>>>> @abnar...@microsoft.com <abnar...@microsoft.com> Abhishek Narain, who >>>>> can provide more detailed information about our roadmap, resources, and >>>>> long-term strategy for supporting this integration. >>>>> >>>>> Thank you again for your patience. We are eager to work alongside the >>>>> community to ensure the success and sustainability of the Microsoft Fabric >>>>> provider. >>>>> >>>>> Best regards, >>>>> Ambika Garg >>>>> >>>>
