Hey Everyone,
I just created a first draft of an Airflow Improvement Proposal for Remote Dag
Fetching https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher
The initial PR is located here:
https://github.com/apache/incubator-airflow/pull/3138
I tried to go through and encapsulate all the various ideas discussed in the
PR, but I’m sure I missed some. As well, there were a number of design
decisions I felt should be further discussed by the community. I would really
appreciate people’s feedback so we can start working towards a fully fleshed
out design. I think the ability to remotely fetch dags, and to stop crawling a
file system is the next big improvement for Airflow.
Thanks!
Ian Davison
The content of this e-mail message and any attached files transmitted with it
are to be treated as confidential information and are intended solely for the
use of the individual or entity to whom they are addressed. If the reader of
this e-mail is not the intended recipient or his or her authorized agent, the
reader is hereby notified that any reproduction, distribution, or disclosure of
this e-mail is prohibited. If you have received this e-mail in error, please
notify the sender by replying to this message and delete this e-mail
immediately.***1010data, Inc. and its affiliates ("1010data") are not
responsible for any advice concerning the use of its software or services or
the data manipulated by its software, except to the extent 1010data has
specifically undertaken such responsibility in a validly binding contract.
1010data shall have no responsibility for any decision concerning the
appropriate method of use or application of its software or services or the
data therein in connection with any transaction. Any decision concerning how,
where and when to use this facility remains the sole responsibility of the
user. NO AGREEMENTS MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL
EXCHANGE REGARDLESS OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS
MUST BE IN THE FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE
OFFICER, PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF
PEOPLE OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.
Hey Everyone,
I just created a first draft of an Airflow Improvement Proposal for Remote Dag
Fetching https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher
The initial PR is located here:
https://github.com/apache/incubator-airflow/pull/3138
I tried to go through and encapsulate all the various ideas discussed in the
PR, but I’m sure I missed some. As well, there were a number of design
decisions I felt should be further discussed by the community. I would really
appreciate people’s feedback so we can start working towards a fully fleshed
out design. I think the ability to remotely fetch dags, and to stop crawling a
file system is the next big improvement for Airflow.
Thanks!
Ian Davison
The content of this e-mail message and any attached files transmitted with it
are to be treated as confidential information and are intended solely for the
use of the individual or entity to whom they are addressed. If the reader of
this e-mail is not the intended recipient or his or her authorized agent, the
reader is hereby notified that any reproduction, distribution, or disclosure of
this e-mail is prohibited. If you have received this e-mail in error, please
notify the sender by replying to this message and delete this e-mail
immediately.***1010data, Inc. and its affiliates ("1010data") are not
responsible for any advice concerning the use of its software or services or
the data manipulated by its software, except to the extent 1010data has
specifically undertaken such responsibility in a validly binding contract.
1010data shall have no responsibility for any decision concerning the
appropriate method of use or application of its software or services or the
data therein in connection with any transaction. Any decision concerning how,
where and when to use this facility remains the sole responsibility of the
user. NO AGREEMENTS MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL
EXCHANGE REGARDLESS OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS
MUST BE IN THE FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE
OFFICER, PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF
PEOPLE OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.
