[jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken
[ https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714687#comment-16714687 ] jack commented on AIRFLOW-1026: --- SQLite can not be used with LocalExecutor > connection string using _cmd tin airflow.cfg is broken > -- > > Key: AIRFLOW-1026 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1026 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.8.0 >Reporter: Harish Singh >Priority: Critical > Fix For: 1.8.0 > > > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > I am expectiing configure.py to be invoked. > But it just throws: > "cannot use sqlite with the LocalExecutor" > The connection string that my script "configure.py" would return is something > like this: > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > But after debugging, I found that, my script is not getting invoked at all. > This is my airflow.cfg: > executor = LocalExecutor > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > sql_alchemy_pool_size = 5 > sql_alchemy_pool_recycle = 3600 > I tried not using the script and directly hardcoding the conn_url > sql_alchemy_conn = > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > It works. > But there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken
[ https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990257#comment-15990257 ] Gavriel Fleischer commented on AIRFLOW-1026: I see the same problem with the sqlite error. I tried to override it: sql_alchemy_conn = but then I get this error: sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '' > connection string using _cmd tin airflow.cfg is broken > -- > > Key: AIRFLOW-1026 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1026 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: Airflow 1.8 >Reporter: Harish Singh >Priority: Critical > Fix For: Airflow 1.8 > > > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > I am expectiing configure.py to be invoked. > But it just throws: > "cannot use sqlite with the LocalExecutor" > The connection string that my script "configure.py" would return is something > like this: > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > But after debugging, I found that, my script is not getting invoked at all. > This is my airflow.cfg: > executor = LocalExecutor > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > sql_alchemy_pool_size = 5 > sql_alchemy_pool_recycle = 3600 > I tried not using the script and directly hardcoding the conn_url > sql_alchemy_conn = > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > It works. > But there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken
[ https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979437#comment-15979437 ] Marc Weil commented on AIRFLOW-1026: I'm having the same issue as well, and sadly this will prevent me from upgrading to 1.8.0 until it's resolved. > connection string using _cmd tin airflow.cfg is broken > -- > > Key: AIRFLOW-1026 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1026 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: Airflow 1.8 >Reporter: Harish Singh >Priority: Critical > Fix For: Airflow 1.8 > > > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > I am expectiing configure.py to be invoked. > But it just throws: > "cannot use sqlite with the LocalExecutor" > The connection string that my script "configure.py" would return is something > like this: > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > But after debugging, I found that, my script is not getting invoked at all. > This is my airflow.cfg: > executor = LocalExecutor > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > sql_alchemy_pool_size = 5 > sql_alchemy_pool_recycle = 3600 > I tried not using the script and directly hardcoding the conn_url > sql_alchemy_conn = > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > It works. > But there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1026) connection string using _cmd tin airflow.cfg is broken
[ https://issues.apache.org/jira/browse/AIRFLOW-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947910#comment-15947910 ] Marty Woodlee commented on AIRFLOW-1026: I am having the same issue. I believe this is happening because the Python {{ConfigParser}} behaves in an additive/"merging" fashion when multiple configs are read, and at https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L524, the built-in {{DEFAULT_CONFIG}} is read before the user's _actual_ config is loaded up at https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L784. Since the default config _does_ contain a {{sql_alchemy_conn}} entry (which points to a local SQLite DB), the config code never tries to evaluate {{sql_alchemy_conn_cmd}} since doing so takes lower precedence (see https://github.com/apache/incubator-airflow/blob/1.8.0/airflow/configuration.py#L592-L605). At first glance I'm not sure any of the {{\_cmd}} options have _ever_ worked? My first instinct is that probably the {{DEFAULT_CONFIG}} should only ever get read if it is known that there is no user-supplied config yet. May need thought, though, if anyone is actually reliant upon that "fall-back" behavior? (Let's hope not.) > connection string using _cmd tin airflow.cfg is broken > -- > > Key: AIRFLOW-1026 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1026 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: Airflow 1.8 >Reporter: Harish Singh >Priority: Critical > Fix For: Airflow 1.8 > > > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > I am expectiing configure.py to be invoked. > But it just throws: > "cannot use sqlite with the LocalExecutor" > The connection string that my script "configure.py" would return is something > like this: > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > But after debugging, I found that, my script is not getting invoked at all. > This is my airflow.cfg: > executor = LocalExecutor > sql_alchemy_conn_cmd = python ./pipeline/dags/configure.py > sql_alchemy_pool_size = 5 > sql_alchemy_pool_recycle = 3600 > I tried not using the script and directly hardcoding the conn_url > sql_alchemy_conn = > mysql+mysqldb://username:**@mysqlhostname:3306/airflowdbname > It works. > But there is a regression bug if somebody wants to use "sql_alchemy_conn_cmd" -- This message was sent by Atlassian JIRA (v6.3.15#6346)