Just in case - please watch the devlist for the announcement of the "SIG multitenancy" group if it slips my mind.
On Thu, Aug 25, 2022 at 1:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Cool. I will make sure to include you ! I think this is something that > will happen in September, The holiday period is not the best to organize it. > > On Thu, Aug 25, 2022 at 5:50 AM Mocheng Guo <gmca...@gmail.com> wrote: > >> My use case needs automation and security: those are the two key >> requirements and does not have to be REST API if there is another way that >> DAGs could be submitted to a cloud storage securely. Sure I would >> appreciate it if you could include me when organizing AIP-1 related >> meetings. Kerberos is a ticket based system in which a ticket has a limited >> lifetime. Using kerberos, a workload could be authenticated before >> persistence so that Airflow uses its kerberos keytab to execute, which is >> similar to the current implementation in worker, another possible scenarios >> is a persisted workload needs to include a kerberos renewable TGT to be >> used by Airflow worker, but this is more complex and I would be happy to >> discuss more in meetings. I will draft a more detailed document for review. >> >> thanks >> Mocheng >> >> >> On Thu, Aug 18, 2022 at 1:19 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> None of those requirements are supported by Airflow. And opening REST >>> API does not solve the authentication use case you mentioned. >>> >>> This is a completely new requirement you have - basically what you want >>> is workflow identity and it should be rather independent from the way DAG >>> is submitted. It would require to attach some kind of identity and >>> signaturea and some way of making sure that the DAG has not been tampered >>> with, in a way that the worker could use the identity when executing the >>> workload and be sure that no-one else modified the DAG - including any of >>> the files that the DAG uses. This is an interesting case but it has nothing >>> to do with using or not the REST API. REST API alone will not give you the >>> user identity guarantees that you need here. The distributed nature of >>> Airflow basically requires such workflow identity has to be provided by >>> cryptographic signatures and verifying the integrity of the DAG rather than >>> basing it on REST API authentication. >>> >>> BTW. We do support already Kerberos authentication for some of our >>> operators but identity is necessarily per instance executing the workload - >>> not the user submitting the DAG. >>> >>> This could be one of the improvement proposals that could in the future >>> become a sub-AIP or AIP-1 (Improve Airflow Security). if you are >>> interested in leading and proposing such an AIP i will be soon (a month or >>> so) re-establishing #sig-multitenancy meetings (see AIP-1 for recordings >>> and minutes of previous meetings). We already have AiP-43 and AIP-44 >>> approved there (and AIP-43 close to completion) and the next steps should >>> be introducing fine graines security layer to executing the workloads. >>> Adding workload identity might be part of it. If you would like to work on >>> that - you are most welcome. It means to prepare and discuss proposals, get >>> consensus of involved parties, leading it to a vote and finally >>> implementing it. >>> >>> J >>> >>> czw., 18 sie 2022, 02:44 użytkownik Mocheng Guo <gmca...@gmail.com> >>> napisał: >>> >>>> >> Could you please elaborate why this would be a problem to use those >>>> (really good for file pushing) APIs ? >>>> >>>> Submitting DAGs directly to cloud storage API does help for some part >>>> of the use case requirement, but cloud storage does not provide the >>>> security a data warehouse needs. A typical auth model supported in data >>>> warehouse is Kerberos, and a data warehouse provides limited view to a >>>> kerberos user with authorization rules. We need users to submit DAGs with >>>> identities supported by the data warehouse, so that Apache Spark jobs will >>>> be executed as the kerberos user who submits a DAG which in turns decide >>>> what data can be processed, there may also be need to handle impersonation, >>>> so there needs to be an additional layer to handle data warehouse auth e.g. >>>> kerberos. >>>> >>>> Assuming dags are already inside the cloud storage, and I think >>>> AIP-5/20 would work better than the current mono repo model if it could >>>> support better flexibility and less latency, and I would be very interested >>>> to be part of the design and implementation. >>>> >>>> >>>> On Fri, Aug 12, 2022 at 10:56 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>>> First appreciate all for your valuable feedback. Airflow by design has >>>>> to accept code, both Tomasz and Constance's examples let me think that the >>>>> security judgement should be on the actual DAGs rather than how DAGs are >>>>> accepted or a process itself. To expand a little bit more on another >>>>> example, say another service provides an API which can be invoked by its >>>>> clients the service validates user inputs e.g. SQL and generates Airflow >>>>> DAGs which use the validated operators/macros. Those DAGs are safe to be >>>>> pushed through the API. There are certainly cases that DAGs may not be >>>>> safe, e.g the API service on public cloud with shared tenants with no >>>>> knowledge how DAGs are generated, in such cases the API service can access >>>>> control the identity or even reject all calls when considered unsafe. >>>>> Please let me know if the example makes sense, and if there is a common >>>>> interest, having an Airflow native write path would benefit the community >>>>> instead of each building its own solution. >>>>> >>>>> > You seem to repeat more of the same. This is exactly what we want to >>>>> avoid. IF you can push a code over API you can push Any Code. And >>>>> precisely >>>>> the "Access Control" you mentioned or rejecting the call when "considering >>>>> code unsafe" those are the decisions we already deliberately decided we do >>>>> not want Airflow REST API to make. Whether the code it's generated or not >>>>> does not matter because Airflow has no idea whatsoever if it has been >>>>> manipulated with, between the time it was generated and pushed. The only >>>>> way Airflow can know that the code is not manipulated with is when it >>>>> generates DAG code on its own based on a declarative input. The limit is >>>>> to >>>>> push declarative information only. You CANNOT push code via the REST API. >>>>> This is out of the question. The case is closed. >>>>> >>>>> The middle loop usually happens on a Jupyter notebook, it needs to >>>>> change data/features used by model frequently which in turn leads to >>>>> Airflow DAG updates, do you mind elaborate how to automate the changes >>>>> inside a notebook and programmatically submitting DAGs through git+CI/CD >>>>> while giving user quick feedback? I understand git+ci/cd is technically >>>>> possible but the overhead involved is a major reason users rejecting >>>>> Airflow for other alternative solutions, e.g. git repo requires manual >>>>> approval even if DAGs can be programmatically submitted, and CI/CD are >>>>> slow >>>>> offline processes with large repo. >>>>> >>>>> Case 2 is actually (if you attempt to read my article I posted above, >>>>> it's written there) the case where shared volume could still be used and >>>>> are bette. This why it's great that Airflow supports multiple DAG syncing >>>>> solutions because your "middle" environment does not have to have git sync >>>>> as it is not "production' (unless you want to mix development with testing >>>>> that is, which is terrible, terrible idea). >>>>> >>>>> Your data science for middle ground does: >>>>> >>>>> a) cp my_dag.py "/my_midle_volume_shared_and_mounted_locally". - if >>>>> you use shared volume of some sort (NFS/EFS etc.) >>>>> b) aws s3 cp my_dag.py "s3://my-midle-testing-bucket/" - if your dags >>>>> are on S3 and synced using s3-sync >>>>> c) gsutil cp my_dag.py "gs://my-bucket" - if your dags are on GCS and >>>>> synced using s3-sync >>>>> >>>>> Those are excellent "File push" apis. They do the job. I cannot >>>>> imagine why the middle-loop person might have a problem with using them. >>>>> All of that can also be fully automated - they all have nice Python and >>>>> other language APIs so you can even make the IDE run those commands >>>>> automatically on every save if you want. >>>>> >>>>> Could you please elaborate why this would be a problem to use those >>>>> (really good for file pushing) APIs ? >>>>> >>>>> J. >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Aug 12, 2022 at 6:20 PM Mocheng Guo <gmca...@gmail.com> wrote: >>>>> >>>>>> First appreciate all for your valuable feedback. Airflow by design >>>>>> has to accept code, both Tomasz and Constance's examples let me think >>>>>> that >>>>>> the security judgement should be on the actual DAGs rather than how DAGs >>>>>> are accepted or a process itself. To expand a little bit more on another >>>>>> example, say another service provides an API which can be invoked by its >>>>>> clients the service validates user inputs e.g. SQL and generates Airflow >>>>>> DAGs which use the validated operators/macros. Those DAGs are safe to be >>>>>> pushed through the API. There are certainly cases that DAGs may not be >>>>>> safe, e.g the API service on public cloud with shared tenants with no >>>>>> knowledge how DAGs are generated, in such cases the API service can >>>>>> access >>>>>> control the identity or even reject all calls when considered unsafe. >>>>>> Please let me know if the example makes sense, and if there is a common >>>>>> interest, having an Airflow native write path would benefit the community >>>>>> instead of each building its own solution. >>>>>> >>>>>> Hi Xiaodong/Jarek, for your suggestion let me elaborate on a use >>>>>> case, here are three loops a data scientist is doing to develop a machine >>>>>> learning model: >>>>>> - inner loop: iterates on the model locally. >>>>>> - middle loop: iterate the model on a remote cluster with production >>>>>> data, say it's using Airflow DAGs behind the scenes. >>>>>> - outer loop: done with iteration and publish the model on production. >>>>>> The middle loop usually happens on a Jupyter notebook, it needs to >>>>>> change data/features used by model frequently which in turn leads to >>>>>> Airflow DAG updates, do you mind elaborate how to automate the changes >>>>>> inside a notebook and programmatically submitting DAGs through git+CI/CD >>>>>> while giving user quick feedback? I understand git+ci/cd is technically >>>>>> possible but the overhead involved is a major reason users rejecting >>>>>> Airflow for other alternative solutions, e.g. git repo requires manual >>>>>> approval even if DAGs can be programmatically submitted, and CI/CD are >>>>>> slow >>>>>> offline processes with large repo. >>>>>> >>>>>> Such use case is pretty common for data scientists, and a better >>>>>> **online** service model would help open up more possibilities for >>>>>> Airflow >>>>>> and its users, as additional layers providing more values(like Constance >>>>>> mentioned enable users with no engineering or airflow domain knowledge to >>>>>> use Airflow) could be built on top of Airflow which remains as a lower >>>>>> level orchestration engine. >>>>>> >>>>>> thanks >>>>>> Mocheng >>>>>> >>>>>> >>>>>> On Thu, Aug 11, 2022 at 10:46 PM Jarek Potiuk <ja...@potiuk.com> >>>>>> wrote: >>>>>> >>>>>>> I really like the Idea of Tomek. >>>>>>> >>>>>>> If we ever go (which is not unlikely) - some "standard" declarative >>>>>>> way of describing DAGs, all my security, packaging concerns are gone - >>>>>>> and >>>>>>> submitting such declarative DAG via API is quite viable. Simply >>>>>>> submitting >>>>>>> a Python code this way is a no-go for me :). Such Declarative DAG could >>>>>>> be >>>>>>> just stored in the DB and scheduled and executed using only >>>>>>> "declaration" >>>>>>> from the DB - without ever touching the DAG "folder" and without >>>>>>> allowing >>>>>>> the user to submit any executable code this way. All the code to execute >>>>>>> would already have to be in Airflow already in this case. >>>>>>> >>>>>>> And I very much agree also that this case can be solved with Git. I >>>>>>> think we are generally undervaluing the role Git plays for DAG >>>>>>> distribution >>>>>>> of Airflow. >>>>>>> >>>>>>> I think when the user feels the need (I very much understand the >>>>>>> need Constance) to submit the DAG via API, rather than adding the >>>>>>> option >>>>>>> of submitting the DAG code via "Airflow REST API", we should simply >>>>>>> answer >>>>>>> this: >>>>>>> >>>>>>> *Use Git and git sync. Then "Git Push" then becomes the standard >>>>>>> "API" you wanted to push the code.* >>>>>>> >>>>>>> This has all the flexibility you need, it has integration with Pull >>>>>>> Request, CI workflows, keeps history etc.etc. When we tell people "Use >>>>>>> Git" >>>>>>> - we have ALL of that and more for free. Standing on the shoulders of >>>>>>> giants. >>>>>>> If we start thinking about integration of code push via our own API >>>>>>> - we basically start the journey of rewriting Git as eventually we will >>>>>>> have to support those cases. This makes absolutely no sense for me. >>>>>>> >>>>>>> I even start to think that we should make "git sync" a separate (and >>>>>>> much more viable) option that is pretty much the "main recommendation" >>>>>>> for >>>>>>> Airflow. rather than "yet another option among shared folders and baked >>>>>>> in >>>>>>> DAGs" case. >>>>>>> >>>>>>> I recently even wrote my thoughts about it in this post: "Shared >>>>>>> Volumes in Airflow - the good, the bad and the ugly": >>>>>>> https://medium.com/apache-airflow/shared-volumes-in-airflow-the-good-the-bad-and-the-ugly-22e9f681afca >>>>>>> which has much more details on why I think so. >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 11, 2022 at 8:43 PM Constance Martineau >>>>>>> <consta...@astronomer.io.invalid> wrote: >>>>>>> >>>>>>>> I understand the security concerns, and generally agree, but as a >>>>>>>> regular user I always wished we could upload DAG files via an API. It >>>>>>>> opens >>>>>>>> the door to have an upload button, which would be nice. It would make >>>>>>>> Airflow a lot more accessible to non-engineering types. >>>>>>>> >>>>>>>> I love the idea of implementing a manual review option in >>>>>>>> conjunction with some sort of hook (similar to Airflow cluster >>>>>>>> policies) >>>>>>>> would be a good middle ground. An administrator could use that hook to >>>>>>>> do >>>>>>>> checks against DAGs or run security scanners, and decide whether or >>>>>>>> not to >>>>>>>> implement a review requirement. >>>>>>>> >>>>>>>> On Thu, Aug 11, 2022 at 1:54 PM Tomasz Urbaszek < >>>>>>>> turbas...@apache.org> wrote: >>>>>>>> >>>>>>>>> In general I second what XD said. CI/CD feels better than sending >>>>>>>>> DAG files over API and the security issues arising from accepting "any >>>>>>>>> python file" are probably quite big. >>>>>>>>> >>>>>>>>> However, I think this proposal can be tightly related to >>>>>>>>> "declarative DAGs". Instead of sending a DAG file, the user would >>>>>>>>> send the >>>>>>>>> DAG definition (operators, inputs, relations) in a predefined format >>>>>>>>> that is not a code. This of course has some limitations like >>>>>>>>> inability to >>>>>>>>> define custom macros, callbacks on the fly but it may be a good >>>>>>>>> compromise. >>>>>>>>> >>>>>>>>> Other thought - if we implement something like "DAG via API" then >>>>>>>>> we should consider adding an option to review DAGs (approval queue >>>>>>>>> etc) to >>>>>>>>> reduce security issues that are mitigated by for example deploying >>>>>>>>> DAGs >>>>>>>>> from git (where we have code review, security scanners etc). >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Tomek >>>>>>>>> >>>>>>>>> On Thu, 11 Aug 2022 at 17:50, Xiaodong Deng <xdd...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Mocheng, >>>>>>>>>> >>>>>>>>>> Please allow me to share a question first: so in your proposal, >>>>>>>>>> the API in your plan is still accepting an Airflow DAG as the >>>>>>>>>> payload (just >>>>>>>>>> binarized or compressed), right? >>>>>>>>>> >>>>>>>>>> If that's the case, I may not be fully convinced: the objectives >>>>>>>>>> in your proposal is about automation & programmatically submitting >>>>>>>>>> DAGs. >>>>>>>>>> These can already be achieved in an efficient way through CI/CD >>>>>>>>>> practice + >>>>>>>>>> a centralized place to manage your DAGs (e.g. a Git Repo to host the >>>>>>>>>> DAG >>>>>>>>>> files). >>>>>>>>>> >>>>>>>>>> As you are already aware, allowing this via API adds additional >>>>>>>>>> security concern, and I would doubt if that "breaks even". >>>>>>>>>> >>>>>>>>>> Kindly let me know if I have missed anything or misunderstood >>>>>>>>>> your proposal. Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> XD >>>>>>>>>> ---------------------------------------------------------------- >>>>>>>>>> (This is not a contribution) >>>>>>>>>> >>>>>>>>>> On Wed, Aug 10, 2022 at 1:46 AM Mocheng Guo <gmca...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Everyone, >>>>>>>>>>> >>>>>>>>>>> I have an enhancement proposal for the REST API service. This is >>>>>>>>>>> based on the observations that Airflow users want to be able to >>>>>>>>>>> access >>>>>>>>>>> Airflow more easily as a platform service. >>>>>>>>>>> >>>>>>>>>>> The motivation comes from the following use cases: >>>>>>>>>>> 1. Users like data scientists want to iterate over data quickly >>>>>>>>>>> with interactive feedback in minutes, e.g. managing data pipelines >>>>>>>>>>> inside >>>>>>>>>>> Jupyter Notebook while executing them in a remote airflow cluster. >>>>>>>>>>> 2. Services targeting specific audiences can generate DAGs based >>>>>>>>>>> on inputs like user command or external triggers, and they want to >>>>>>>>>>> be able >>>>>>>>>>> to submit DAGs programmatically without manual intervention. >>>>>>>>>>> >>>>>>>>>>> I believe such use cases would help promote Airflow usability >>>>>>>>>>> and gain more customer popularity. The existing DAG repo brings >>>>>>>>>>> considerable overhead for such scenarios, a shared repo requires >>>>>>>>>>> offline >>>>>>>>>>> processes and can be slow to rollout. >>>>>>>>>>> >>>>>>>>>>> The proposal aims to provide an alternative where a DAG can be >>>>>>>>>>> transmitted online and here are some key points: >>>>>>>>>>> 1. A DAG is packaged individually so that it can be >>>>>>>>>>> distributable over the network. For example, a DAG may be a >>>>>>>>>>> serialized >>>>>>>>>>> binary or a zip file. >>>>>>>>>>> 2. The Airflow REST API is the ideal place to talk with the >>>>>>>>>>> external world. The API would provide a generic interface to accept >>>>>>>>>>> DAG >>>>>>>>>>> artifacts and should be extensible to support different artifact >>>>>>>>>>> formats if >>>>>>>>>>> needed. >>>>>>>>>>> 3. DAG persistence needs to be implemented since they are not >>>>>>>>>>> part of the DAG repository. >>>>>>>>>>> 4. Same behavior for DAGs supported in API vs those defined in >>>>>>>>>>> the repo, i.e. users write DAGs in the same syntax, and its >>>>>>>>>>> scheduling, >>>>>>>>>>> execution, and web server UI should behave the same way. >>>>>>>>>>> >>>>>>>>>>> Since DAGs are written as code, running arbitrary code inside >>>>>>>>>>> Airflow may pose high security risks. Here are a few proposals to >>>>>>>>>>> stop the >>>>>>>>>>> security breach: >>>>>>>>>>> 1. Accept DAGs only from trusted parties. Airflow already >>>>>>>>>>> supports pluggable authentication modules where strong >>>>>>>>>>> authentication such >>>>>>>>>>> as Kerberos can be used. >>>>>>>>>>> 2. Execute DAG code as the API identity, i.e. A DAG created >>>>>>>>>>> through the API service will have run_as_user set to be the API >>>>>>>>>>> identity. >>>>>>>>>>> 3. To enforce data access control on DAGs, the API identity >>>>>>>>>>> should also be used to access the data warehouse. >>>>>>>>>>> >>>>>>>>>>> We shared a demo based on a prototype implementation in the >>>>>>>>>>> summit and some details are described in this ppt >>>>>>>>>>> <https://drive.google.com/file/d/1luDGvWRA-hwn2NjPoobis2SL4_UNYfcM/view>, >>>>>>>>>>> and would love to get feedback and comments from the community >>>>>>>>>>> about this >>>>>>>>>>> initiative. >>>>>>>>>>> >>>>>>>>>>> thanks >>>>>>>>>>> Mocheng >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Constance Martineau >>>>>>>> Product Manager >>>>>>>> >>>>>>>> Email: consta...@astronomer.io >>>>>>>> Time zone: US Eastern (EST UTC-5 / EDT UTC-4) >>>>>>>> >>>>>>>> >>>>>>>> <https://www.astronomer.io/> >>>>>>>> >>>>>>>>
