Exactly that Ash! This awesome community never fails to impress. Thanks
folks!
On Tue, Sep 25, 2018 at 3:29 AM Ash Berlin-Taylor wrote:
>
> > On 24 Sep 2018, at 23:12, Alex Tronchin-James 949-412-7220
> <(949)%20412-7220> wrote:
> >
> > Re: [Brian Greene] "How does filename matter? Frankly I w
> On 24 Sep 2018, at 23:12, Alex Tronchin-James 949-412-7220
> wrote:
>
> Re: [Brian Greene] "How does filename matter? Frankly I wish the filename
> was REQUIRED to be the dag name so people would quit confusing themselves
> by mismatching them !"
>
> FWIW in the Facebook predecessor to airf
I think decoupling dag_id and display name could be confusing and
cumbersome. As for readme, DAG already has a field called description which
I think is close to what Alex is describing (I believe it is displayed by
the UI).
On Mon, Sep 24, 2018 at 3:12 PM Alex Tronchin-James 949-412-7220 <
alex.n
Re: [Brian Greene] "How does filename matter? Frankly I wish the filename
was REQUIRED to be the dag name so people would quit confusing themselves
by mismatching them !"
FWIW in the Facebook predecessor to airflow, the file path/name WAS the dag
name. E.g. if your dag resided in best_team/new_pr
Sounds like this may be a good candidate for an Airflow Improvement Plan...
On 20 September 2018 at 22:45, Brian Greene
wrote:
> Prior to using airflow for much, on first inspection, I think I may have
> agreed with you.
>
> After a bit of use I’d agree with Fokko and others - this isn’t really
Prior to using airflow for much, on first inspection, I think I may have agreed
with you.
After a bit of use I’d agree with Fokko and others - this isn’t really a
problem, and separating them seems to do more harm than good related to
deployment.
I was gonna stop there, but why?
You can add
I like the dag_id for both the name and as an unique identifier. If you
change the dag in such a way, that it deserves a new name, you probably
want to create a new dag anyway. If you want to give some additional
context, you can use the description field:
https://github.com/apache/incubator-airflo
I'm personally against having some kind of auto-increment numeric ID for
DAGs. While this makes a lot of sense for systems where creation is a
database activity (like a POST request), in Airflow, DAG creation is
actually a code ship activity. There are all kinds of complex scenarios
around that:
-
Hi,
though this could have been explained on Jira I think this should be discussed
first.
The problem:
Airflow mixes DAG name with id. It uses same filed for both purposes.
I assume that most of you use the dag_id to describe what the DAG actually does.
For example:
dag = DAG(
dag_id='cost_
