Cool. I will make sure to include you ! I think this is something that will happen in September, The holiday period is not the best to organize it.
On Thu, Aug 25, 2022 at 5:50 AM Mocheng Guo <gmca...@gmail.com> wrote: > My use case needs automation and security: those are the two key > requirements and does not have to be REST API if there is another way that > DAGs could be submitted to a cloud storage securely. Sure I would > appreciate it if you could include me when organizing AIP-1 related > meetings. Kerberos is a ticket based system in which a ticket has a limited > lifetime. Using kerberos, a workload could be authenticated before > persistence so that Airflow uses its kerberos keytab to execute, which is > similar to the current implementation in worker, another possible scenarios > is a persisted workload needs to include a kerberos renewable TGT to be > used by Airflow worker, but this is more complex and I would be happy to > discuss more in meetings. I will draft a more detailed document for review. > > thanks > Mocheng > > > On Thu, Aug 18, 2022 at 1:19 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> None of those requirements are supported by Airflow. And opening REST API >> does not solve the authentication use case you mentioned. >> >> This is a completely new requirement you have - basically what you want >> is workflow identity and it should be rather independent from the way DAG >> is submitted. It would require to attach some kind of identity and >> signaturea and some way of making sure that the DAG has not been tampered >> with, in a way that the worker could use the identity when executing the >> workload and be sure that no-one else modified the DAG - including any of >> the files that the DAG uses. This is an interesting case but it has nothing >> to do with using or not the REST API. REST API alone will not give you the >> user identity guarantees that you need here. The distributed nature of >> Airflow basically requires such workflow identity has to be provided by >> cryptographic signatures and verifying the integrity of the DAG rather than >> basing it on REST API authentication. >> >> BTW. We do support already Kerberos authentication for some of our >> operators but identity is necessarily per instance executing the workload - >> not the user submitting the DAG. >> >> This could be one of the improvement proposals that could in the future >> become a sub-AIP or AIP-1 (Improve Airflow Security). if you are >> interested in leading and proposing such an AIP i will be soon (a month or >> so) re-establishing #sig-multitenancy meetings (see AIP-1 for recordings >> and minutes of previous meetings). We already have AiP-43 and AIP-44 >> approved there (and AIP-43 close to completion) and the next steps should >> be introducing fine graines security layer to executing the workloads. >> Adding workload identity might be part of it. If you would like to work on >> that - you are most welcome. It means to prepare and discuss proposals, get >> consensus of involved parties, leading it to a vote and finally >> implementing it. >> >> J >> >> czw., 18 sie 2022, 02:44 użytkownik Mocheng Guo <gmca...@gmail.com> >> napisał: >> >>> >> Could you please elaborate why this would be a problem to use those >>> (really good for file pushing) APIs ? >>> >>> Submitting DAGs directly to cloud storage API does help for some part of >>> the use case requirement, but cloud storage does not provide the security a >>> data warehouse needs. A typical auth model supported in data warehouse is >>> Kerberos, and a data warehouse provides limited view to a kerberos user >>> with authorization rules. We need users to submit DAGs with identities >>> supported by the data warehouse, so that Apache Spark jobs will be executed >>> as the kerberos user who submits a DAG which in turns decide what data can >>> be processed, there may also be need to handle impersonation, so there >>> needs to be an additional layer to handle data warehouse auth e.g. >>> kerberos. >>> >>> Assuming dags are already inside the cloud storage, and I think AIP-5/20 >>> would work better than the current mono repo model if it could support >>> better flexibility and less latency, and I would be very interested to be >>> part of the design and implementation. >>> >>> >>> On Fri, Aug 12, 2022 at 10:56 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> First appreciate all for your valuable feedback. Airflow by design has >>>> to accept code, both Tomasz and Constance's examples let me think that the >>>> security judgement should be on the actual DAGs rather than how DAGs are >>>> accepted or a process itself. To expand a little bit more on another >>>> example, say another service provides an API which can be invoked by its >>>> clients the service validates user inputs e.g. SQL and generates Airflow >>>> DAGs which use the validated operators/macros. Those DAGs are safe to be >>>> pushed through the API. There are certainly cases that DAGs may not be >>>> safe, e.g the API service on public cloud with shared tenants with no >>>> knowledge how DAGs are generated, in such cases the API service can access >>>> control the identity or even reject all calls when considered unsafe. >>>> Please let me know if the example makes sense, and if there is a common >>>> interest, having an Airflow native write path would benefit the community >>>> instead of each building its own solution. >>>> >>>> > You seem to repeat more of the same. This is exactly what we want to >>>> avoid. IF you can push a code over API you can push Any Code. And precisely >>>> the "Access Control" you mentioned or rejecting the call when "considering >>>> code unsafe" those are the decisions we already deliberately decided we do >>>> not want Airflow REST API to make. Whether the code it's generated or not >>>> does not matter because Airflow has no idea whatsoever if it has been >>>> manipulated with, between the time it was generated and pushed. The only >>>> way Airflow can know that the code is not manipulated with is when it >>>> generates DAG code on its own based on a declarative input. The limit is to >>>> push declarative information only. You CANNOT push code via the REST API. >>>> This is out of the question. The case is closed. >>>> >>>> The middle loop usually happens on a Jupyter notebook, it needs to >>>> change data/features used by model frequently which in turn leads to >>>> Airflow DAG updates, do you mind elaborate how to automate the changes >>>> inside a notebook and programmatically submitting DAGs through git+CI/CD >>>> while giving user quick feedback? I understand git+ci/cd is technically >>>> possible but the overhead involved is a major reason users rejecting >>>> Airflow for other alternative solutions, e.g. git repo requires manual >>>> approval even if DAGs can be programmatically submitted, and CI/CD are slow >>>> offline processes with large repo. >>>> >>>> Case 2 is actually (if you attempt to read my article I posted above, >>>> it's written there) the case where shared volume could still be used and >>>> are bette. This why it's great that Airflow supports multiple DAG syncing >>>> solutions because your "middle" environment does not have to have git sync >>>> as it is not "production' (unless you want to mix development with testing >>>> that is, which is terrible, terrible idea). >>>> >>>> Your data science for middle ground does: >>>> >>>> a) cp my_dag.py "/my_midle_volume_shared_and_mounted_locally". - if you >>>> use shared volume of some sort (NFS/EFS etc.) >>>> b) aws s3 cp my_dag.py "s3://my-midle-testing-bucket/" - if your dags >>>> are on S3 and synced using s3-sync >>>> c) gsutil cp my_dag.py "gs://my-bucket" - if your dags are on GCS and >>>> synced using s3-sync >>>> >>>> Those are excellent "File push" apis. They do the job. I cannot imagine >>>> why the middle-loop person might have a problem with using them. All of >>>> that can also be fully automated - they all have nice Python and >>>> other language APIs so you can even make the IDE run those commands >>>> automatically on every save if you want. >>>> >>>> Could you please elaborate why this would be a problem to use those >>>> (really good for file pushing) APIs ? >>>> >>>> J. >>>> >>>> >>>> >>>> >>>> On Fri, Aug 12, 2022 at 6:20 PM Mocheng Guo <gmca...@gmail.com> wrote: >>>> >>>>> First appreciate all for your valuable feedback. Airflow by design has >>>>> to accept code, both Tomasz and Constance's examples let me think that the >>>>> security judgement should be on the actual DAGs rather than how DAGs are >>>>> accepted or a process itself. To expand a little bit more on another >>>>> example, say another service provides an API which can be invoked by its >>>>> clients the service validates user inputs e.g. SQL and generates Airflow >>>>> DAGs which use the validated operators/macros. Those DAGs are safe to be >>>>> pushed through the API. There are certainly cases that DAGs may not be >>>>> safe, e.g the API service on public cloud with shared tenants with no >>>>> knowledge how DAGs are generated, in such cases the API service can access >>>>> control the identity or even reject all calls when considered unsafe. >>>>> Please let me know if the example makes sense, and if there is a common >>>>> interest, having an Airflow native write path would benefit the community >>>>> instead of each building its own solution. >>>>> >>>>> Hi Xiaodong/Jarek, for your suggestion let me elaborate on a use case, >>>>> here are three loops a data scientist is doing to develop a machine >>>>> learning model: >>>>> - inner loop: iterates on the model locally. >>>>> - middle loop: iterate the model on a remote cluster with production >>>>> data, say it's using Airflow DAGs behind the scenes. >>>>> - outer loop: done with iteration and publish the model on production. >>>>> The middle loop usually happens on a Jupyter notebook, it needs to >>>>> change data/features used by model frequently which in turn leads to >>>>> Airflow DAG updates, do you mind elaborate how to automate the changes >>>>> inside a notebook and programmatically submitting DAGs through git+CI/CD >>>>> while giving user quick feedback? I understand git+ci/cd is technically >>>>> possible but the overhead involved is a major reason users rejecting >>>>> Airflow for other alternative solutions, e.g. git repo requires manual >>>>> approval even if DAGs can be programmatically submitted, and CI/CD are >>>>> slow >>>>> offline processes with large repo. >>>>> >>>>> Such use case is pretty common for data scientists, and a better >>>>> **online** service model would help open up more possibilities for Airflow >>>>> and its users, as additional layers providing more values(like Constance >>>>> mentioned enable users with no engineering or airflow domain knowledge to >>>>> use Airflow) could be built on top of Airflow which remains as a lower >>>>> level orchestration engine. >>>>> >>>>> thanks >>>>> Mocheng >>>>> >>>>> >>>>> On Thu, Aug 11, 2022 at 10:46 PM Jarek Potiuk <ja...@potiuk.com> >>>>> wrote: >>>>> >>>>>> I really like the Idea of Tomek. >>>>>> >>>>>> If we ever go (which is not unlikely) - some "standard" declarative >>>>>> way of describing DAGs, all my security, packaging concerns are gone - >>>>>> and >>>>>> submitting such declarative DAG via API is quite viable. Simply >>>>>> submitting >>>>>> a Python code this way is a no-go for me :). Such Declarative DAG could >>>>>> be >>>>>> just stored in the DB and scheduled and executed using only "declaration" >>>>>> from the DB - without ever touching the DAG "folder" and without allowing >>>>>> the user to submit any executable code this way. All the code to execute >>>>>> would already have to be in Airflow already in this case. >>>>>> >>>>>> And I very much agree also that this case can be solved with Git. I >>>>>> think we are generally undervaluing the role Git plays for DAG >>>>>> distribution >>>>>> of Airflow. >>>>>> >>>>>> I think when the user feels the need (I very much understand the need >>>>>> Constance) to submit the DAG via API, rather than adding the option of >>>>>> submitting the DAG code via "Airflow REST API", we should simply answer >>>>>> this: >>>>>> >>>>>> *Use Git and git sync. Then "Git Push" then becomes the standard >>>>>> "API" you wanted to push the code.* >>>>>> >>>>>> This has all the flexibility you need, it has integration with Pull >>>>>> Request, CI workflows, keeps history etc.etc. When we tell people "Use >>>>>> Git" >>>>>> - we have ALL of that and more for free. Standing on the shoulders of >>>>>> giants. >>>>>> If we start thinking about integration of code push via our own API - >>>>>> we basically start the journey of rewriting Git as eventually we will >>>>>> have >>>>>> to support those cases. This makes absolutely no sense for me. >>>>>> >>>>>> I even start to think that we should make "git sync" a separate (and >>>>>> much more viable) option that is pretty much the "main recommendation" >>>>>> for >>>>>> Airflow. rather than "yet another option among shared folders and baked >>>>>> in >>>>>> DAGs" case. >>>>>> >>>>>> I recently even wrote my thoughts about it in this post: "Shared >>>>>> Volumes in Airflow - the good, the bad and the ugly": >>>>>> https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca >>>>>> which has much more details on why I think so. >>>>>> >>>>>> J. >>>>>> >>>>>> >>>>>> On Thu, Aug 11, 2022 at 8:43 PM Constance Martineau >>>>>> <consta...@astronomer.io.invalid> wrote: >>>>>> >>>>>>> I understand the security concerns, and generally agree, but as a >>>>>>> regular user I always wished we could upload DAG files via an API. It >>>>>>> opens >>>>>>> the door to have an upload button, which would be nice. It would make >>>>>>> Airflow a lot more accessible to non-engineering types. >>>>>>> >>>>>>> I love the idea of implementing a manual review option in >>>>>>> conjunction with some sort of hook (similar to Airflow cluster policies) >>>>>>> would be a good middle ground. An administrator could use that hook to >>>>>>> do >>>>>>> checks against DAGs or run security scanners, and decide whether or not >>>>>>> to >>>>>>> implement a review requirement. >>>>>>> >>>>>>> On Thu, Aug 11, 2022 at 1:54 PM Tomasz Urbaszek < >>>>>>> turbas...@apache.org> wrote: >>>>>>> >>>>>>>> In general I second what XD said. CI/CD feels better than sending >>>>>>>> DAG files over API and the security issues arising from accepting "any >>>>>>>> python file" are probably quite big. >>>>>>>> >>>>>>>> However, I think this proposal can be tightly related to >>>>>>>> "declarative DAGs". Instead of sending a DAG file, the user would send >>>>>>>> the >>>>>>>> DAG definition (operators, inputs, relations) in a predefined format >>>>>>>> that is not a code. This of course has some limitations like inability >>>>>>>> to >>>>>>>> define custom macros, callbacks on the fly but it may be a good >>>>>>>> compromise. >>>>>>>> >>>>>>>> Other thought - if we implement something like "DAG via API" then >>>>>>>> we should consider adding an option to review DAGs (approval queue >>>>>>>> etc) to >>>>>>>> reduce security issues that are mitigated by for example deploying DAGs >>>>>>>> from git (where we have code review, security scanners etc). >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Tomek >>>>>>>> >>>>>>>> On Thu, 11 Aug 2022 at 17:50, Xiaodong Deng <xdd...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Mocheng, >>>>>>>>> >>>>>>>>> Please allow me to share a question first: so in your proposal, >>>>>>>>> the API in your plan is still accepting an Airflow DAG as the payload >>>>>>>>> (just >>>>>>>>> binarized or compressed), right? >>>>>>>>> >>>>>>>>> If that's the case, I may not be fully convinced: the objectives >>>>>>>>> in your proposal is about automation & programmatically submitting >>>>>>>>> DAGs. >>>>>>>>> These can already be achieved in an efficient way through CI/CD >>>>>>>>> practice + >>>>>>>>> a centralized place to manage your DAGs (e.g. a Git Repo to host the >>>>>>>>> DAG >>>>>>>>> files). >>>>>>>>> >>>>>>>>> As you are already aware, allowing this via API adds additional >>>>>>>>> security concern, and I would doubt if that "breaks even". >>>>>>>>> >>>>>>>>> Kindly let me know if I have missed anything or misunderstood your >>>>>>>>> proposal. Thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> XD >>>>>>>>> ---------------------------------------------------------------- >>>>>>>>> (This is not a contribution) >>>>>>>>> >>>>>>>>> On Wed, Aug 10, 2022 at 1:46 AM Mocheng Guo <gmca...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Everyone, >>>>>>>>>> >>>>>>>>>> I have an enhancement proposal for the REST API service. This is >>>>>>>>>> based on the observations that Airflow users want to be able to >>>>>>>>>> access >>>>>>>>>> Airflow more easily as a platform service. >>>>>>>>>> >>>>>>>>>> The motivation comes from the following use cases: >>>>>>>>>> 1. Users like data scientists want to iterate over data quickly >>>>>>>>>> with interactive feedback in minutes, e.g. managing data pipelines >>>>>>>>>> inside >>>>>>>>>> Jupyter Notebook while executing them in a remote airflow cluster. >>>>>>>>>> 2. Services targeting specific audiences can generate DAGs based >>>>>>>>>> on inputs like user command or external triggers, and they want to >>>>>>>>>> be able >>>>>>>>>> to submit DAGs programmatically without manual intervention. >>>>>>>>>> >>>>>>>>>> I believe such use cases would help promote Airflow usability and >>>>>>>>>> gain more customer popularity. The existing DAG repo brings >>>>>>>>>> considerable >>>>>>>>>> overhead for such scenarios, a shared repo requires offline >>>>>>>>>> processes and >>>>>>>>>> can be slow to rollout. >>>>>>>>>> >>>>>>>>>> The proposal aims to provide an alternative where a DAG can be >>>>>>>>>> transmitted online and here are some key points: >>>>>>>>>> 1. A DAG is packaged individually so that it can be distributable >>>>>>>>>> over the network. For example, a DAG may be a serialized binary or a >>>>>>>>>> zip >>>>>>>>>> file. >>>>>>>>>> 2. The Airflow REST API is the ideal place to talk with the >>>>>>>>>> external world. The API would provide a generic interface to accept >>>>>>>>>> DAG >>>>>>>>>> artifacts and should be extensible to support different artifact >>>>>>>>>> formats if >>>>>>>>>> needed. >>>>>>>>>> 3. DAG persistence needs to be implemented since they are not >>>>>>>>>> part of the DAG repository. >>>>>>>>>> 4. Same behavior for DAGs supported in API vs those defined in >>>>>>>>>> the repo, i.e. users write DAGs in the same syntax, and its >>>>>>>>>> scheduling, >>>>>>>>>> execution, and web server UI should behave the same way. >>>>>>>>>> >>>>>>>>>> Since DAGs are written as code, running arbitrary code inside >>>>>>>>>> Airflow may pose high security risks. Here are a few proposals to >>>>>>>>>> stop the >>>>>>>>>> security breach: >>>>>>>>>> 1. Accept DAGs only from trusted parties. Airflow already >>>>>>>>>> supports pluggable authentication modules where strong >>>>>>>>>> authentication such >>>>>>>>>> as Kerberos can be used. >>>>>>>>>> 2. Execute DAG code as the API identity, i.e. A DAG created >>>>>>>>>> through the API service will have run_as_user set to be the API >>>>>>>>>> identity. >>>>>>>>>> 3. To enforce data access control on DAGs, the API identity >>>>>>>>>> should also be used to access the data warehouse. >>>>>>>>>> >>>>>>>>>> We shared a demo based on a prototype implementation in the >>>>>>>>>> summit and some details are described in this ppt >>>>>>>>>> <https://drive.google.com/file/d/1luDGvWRA-hwn2NjPoobis2SL4_UNYfcM/view>, >>>>>>>>>> and would love to get feedback and comments from the community about >>>>>>>>>> this >>>>>>>>>> initiative. >>>>>>>>>> >>>>>>>>>> thanks >>>>>>>>>> Mocheng >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Constance Martineau >>>>>>> Product Manager >>>>>>> >>>>>>> Email: consta...@astronomer.io >>>>>>> Time zone: US Eastern (EST UTC-5 / EDT UTC-4) >>>>>>> >>>>>>> >>>>>>> <https://www.astronomer.io/> >>>>>>> >>>>>>>
