potiuk commented on issue #29192:
URL: https://github.com/apache/airflow/issues/29192#issuecomment-1406352484

   That goes exactly against the way how Airflow DAG philosophy about 
distributing DAG files is. Airflow REST API is NOT about adding or modifying 
DAGs. It's about managing DAGs that ar distributed using other mechanism. Your 
interface to deal with DAG files is not Airflow REST API. Your API forr that is 
the API of the filesytem  that you are using for distributing DAGS:
   
   * S3 -> use S3 API
   * Shared EFS volume -> use Shared EFS
   * GitSync -> use Git commands
   * etc. etc. 
   
   This is NOT the question of doing post/pre-processing of what happens with 
the remote call over the API. This is also (and most importantly) the question 
of security and separation of concerns. Airlfow's REST API and Webserver have 
very, very low permissions when it comes to producing or manipulating an 
executable Python Code.
   
   Simply speaking, by interfacing with webserver or REST API, it's just 
impossible to modify any code that is submitted by a DAG Author and executed as 
part of tasks, It's not "disallowed" - it's far more than that - it's not 
POSSIBLE, because neither API nor webserver have (in serious deployments) have 
access to the DAG files on the shared volume, git sync. More often than not the 
webserver (due to the fact that it is exposed to external world via http/https 
and security requirements) will be running with it's own sandbox with only 
necessary ports and IP addresses it can communicate with for outwards 
communication - which will make it imposible to directly talk to s3 for example.
   
   The webserver / API server does not only miss credentials to do so, but they 
don't even have shared folders with DAGs mounted. AT ALL.  Being able to delete 
a file, means that you have to have credentials to do so exposed to the 
webserver/api as well as you have to have those files mounted/accessible. 
   
   So what you are asking for is breaking absolutely basic and foundational 
assumptions of Airlfow about security. And it's very straightforward "no-go".
   
   And it makes no sense whatsoever even from your perspective as someone who 
wants to automate this. If you were to write the python code to communicate 
with S3 for example and delete the files, you not only have to implement the 
code to do it and use the right API (S3 for example), but you also need to 
deploy that code with Airflow, and make sure it is integrated with this new 
futnctionality, distribute and (somehow) protect the credentials that are 
needed to do so (possibly to multiple Webserver/API components). If you do it 
directly in the client of yours that originate the "delete" request and does 
such removal directly, you have MUCH simpler job, and also you are not limited 
to using Python. You can for example write a bash script that runs:
   
   ```
   aws s3 rm "generated URL to your dag file"
   ```
   
   as part of your client deletion process. And all the credentials might be 
precisely where you need them and when you can secure them (via oauth or any 
other mechanism) - at the client which runs the deletion request. It's just 
**simpler** for you than wrapping the same code with Airlfow REST API call and 
integrating it with Airflow installation.
   
   
   If you want to delete DAG files, please continue using S3 API to delete your 
if you use S3. Or EFS API if you use EFS. Or "Git commit" if you are using Git 
Sync. Those are excellent APIs for managing files. Far better than REST API can 
ever be. They are very good for automation and you can make sure in your system 
that you have the right, properly secured credentials on the client side that 
originates the delete action. Do not push responsibility for that to Airflow 
webserver/ API server and do not add extra REST layer on top of those. It makes 
no sense.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to