[jira] [Created] (AIRFLOW-3560) Add Sensor that polls until a day of the week
Kaxil Naik created AIRFLOW-3560: --- Summary: Add Sensor that polls until a day of the week Key: AIRFLOW-3560 URL: https://issues.apache.org/jira/browse/AIRFLOW-3560 Project: Apache Airflow Issue Type: New Feature Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 1.10.2 One of the use-case we have is we want to run certain tasks only on Weekends -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3150) Make execution_date a template field in TriggerDagRunOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3150: Fix Version/s: 1.10.2 > Make execution_date a template field in TriggerDagRunOperator > - > > Key: AIRFLOW-3150 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3150 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Kyle Hamlin >Assignee: Kaxil Naik >Priority: Minor > Labels: easy-fix > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-850) Airflow should support a general purpose PythonSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-850. Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4349 > Airflow should support a general purpose PythonSensor > - > > Key: AIRFLOW-850 > URL: https://issues.apache.org/jira/browse/AIRFLOW-850 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.8.0 >Reporter: Daniel Gies >Assignee: Daniel Gies >Priority: Major > Fix For: 1.10.2 > > > Today I found myself trying to use a sensor to postpone execution until data > for the current execution date appeared in a file. It occurred to me that > having a general purpose PythonSensor would allow developers to use the > sensor paradigm with arbitrary code. > We should add a PythonSensor to the core sensors module which takes a > python_callable and optional args like the PythonOperator does. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3557) Various typos
[ https://issues.apache.org/jira/browse/AIRFLOW-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3557. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/4357 > Various typos > - > > Key: AIRFLOW-3557 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3557 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > Fix various typos, checked with > [misspell|https://github.com/client9/misspell]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3398) Google Cloud Spanner instance database query operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3398. - Resolution: Fixed Assignee: (was: Szymon Przedwojski) Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4314 > Google Cloud Spanner instance database query operator > - > > Key: AIRFLOW-3398 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3398 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Reporter: Szymon Przedwojski >Priority: Minor > Fix For: 1.10.2 > > > Creating an operator to enable executing arbitrary SQL in a Transaction in > Cloud Spanner. > https://googleapis.github.io/google-cloud-python/latest/spanner/index.html#executing-arbitrary-sql-in-a-transaction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3500) Make task duration display user friendly
[ https://issues.apache.org/jira/browse/AIRFLOW-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3500: Fix Version/s: (was: 1.10.2) 2.0.0 > Make task duration display user friendly > > > Key: AIRFLOW-3500 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3500 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Affects Versions: 1.10.1 >Reporter: Ofer Zelig >Assignee: Ofer Zelig >Priority: Major > Fix For: 2.0.0 > > > When hovering over a task (in Graph mode), the duration it took is displayed > as a plain number, which doesn't say what the number is (it's actually > seconds). > When you see something like 2716 it's impractical to know how much time it > actually took, unless you can quickly do the math in your head. > Change the display to read days/hours/minutes/seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3447.
-
Resolution: Fixed
Resolved by https://github.com/apache/incubator-airflow/pull/4323
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Assignee: Kaxil Naik
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3521) Airflow Jira Compare script is limited to 50 items
[ https://issues.apache.org/jira/browse/AIRFLOW-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3521. - Resolution: Fixed Fix Version/s: 1.10.2 > Airflow Jira Compare script is limited to 50 items > -- > > Key: AIRFLOW-3521 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3521 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kaxil Naik >Priority: Minor > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3521) Airflow Jira Compare script is limited to 50 items
Kaxil Naik created AIRFLOW-3521: --- Summary: Airflow Jira Compare script is limited to 50 items Key: AIRFLOW-3521 URL: https://issues.apache.org/jira/browse/AIRFLOW-3521 Project: Apache Airflow Issue Type: Improvement Reporter: Kaxil Naik -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-1919.
-
Resolution: Fixed
Fix Version/s: (was: 2.0.0)
1.10.2
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 1.10.2
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik reopened AIRFLOW-1919:
-
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 1.10.2
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722114#comment-16722114
]
Kaxil Naik commented on AIRFLOW-1919:
-
This feature will be available in 1.10.2. Currently it is only available in the
master
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 1.10.2
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1919) Add option to query for DAG runs given a DAG ID
[
https://issues.apache.org/jira/browse/AIRFLOW-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722113#comment-16722113
]
Kaxil Naik commented on AIRFLOW-1919:
-
[~villasv]: This is already available in the documentation at
https://airflow.readthedocs.io/en/latest/cli.html#list_dag_runs
> Add option to query for DAG runs given a DAG ID
> ---
>
> Key: AIRFLOW-1919
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1919
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.8.0
>Reporter: Steen Manniche
>Assignee: Tao Feng
>Priority: Trivial
> Fix For: 2.0.0
>
>
> Having a way to list all DAG runs for a given DAG identifier would be useful
> when trying to get a programmatic overview of running DAGs. Something along
> the lines of
> {code}
> airflow list_runs $DAG_ID
> {code}
> Which would return the running DAGs for {{$DAG_ID}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3444) Expand templated fields in gcs transfer service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3444. - Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4279 > Expand templated fields in gcs transfer service operator > > > Key: AIRFLOW-3444 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3444 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Fix For: 1.10.2 > > > The `S3ToGoogleCloudStorageTransferOperator` should support an explicit > `description` parameter and allow that parameter to be templated. This will > make it easier to set job descriptions, and we'll be able to drop the > `job_kwargs` attribute, which doesn't have any other use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3411) create openfaas hook
[ https://issues.apache.org/jira/browse/AIRFLOW-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3411. - Resolution: Fixed Fix Version/s: 1.10.2 > create openfaas hook > > > Key: AIRFLOW-3411 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3411 > Project: Apache Airflow > Issue Type: Improvement >Reporter: tal >Assignee: tal >Priority: Minor > Fix For: 1.10.2 > > > add openfaas hook to trigger functions > (https://github.com/openfaas/faas) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3310) Google Cloud Spanner deploy / delete operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3310. - Resolution: Fixed Fix Version/s: 1.10.2 > Google Cloud Spanner deploy / delete operators > -- > > Key: AIRFLOW-3310 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3310 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Reporter: Szymon Przedwojski >Assignee: Szymon Przedwojski >Priority: Minor > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reopened AIRFLOW-2770: - > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > Fix For: 1.10.2 > > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2770. - Resolution: Fixed Fix Version/s: 1.10.2 > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > Fix For: 1.10.2 > > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3327: Fix Version/s: 1.10.2 > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Fix For: 1.10.2 > > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718740#comment-16718740 ] Kaxil Naik commented on AIRFLOW-3327: - I have assigned this to myself [~dsdinter] :-) Will work on it and should resolve it by this weekend > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reassigned AIRFLOW-3327: --- Assignee: Kaxil Naik > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2785) Improve context management of mongo hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2785: Fix Version/s: (was: 2.0.0) 1.10.2 > Improve context management of mongo hook > > > Key: AIRFLOW-2785 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2785 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Andy Cooper >Assignee: Andy Cooper >Priority: Minor > Fix For: 1.10.2 > > > Add __enter__ and __exit__ methods to improve context management of mongohook -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3484) The KubernetesExecutor is overly verbose and this can kill schedulers.
[ https://issues.apache.org/jira/browse/AIRFLOW-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3484: Fix Version/s: (was: 2.0.0) 1.10.2 > The KubernetesExecutor is overly verbose and this can kill schedulers. > -- > > Key: AIRFLOW-3484 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3484 > Project: Apache Airflow > Issue Type: Bug > Components: kubernetes >Affects Versions: 1.10.1 >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Major > Fix For: 1.10.2 > > > There are two log lines in the k8sexecutor that can cause schedulers to crash > just by their sheer verbosity. This PR will switch these lines to debug as to > not mess with normal workflows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3438) BigQueryOperator should default udf_config to None instead of false.
[
https://issues.apache.org/jira/browse/AIRFLOW-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3438.
-
Resolution: Fixed
Resolved by https://github.com/apache/incubator-airflow/pull/4274
> BigQueryOperator should default udf_config to None instead of false.
>
>
> Key: AIRFLOW-3438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3438
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Reporter: Luka Draksler
>Assignee: Kaxil Naik
>Priority: Major
> Labels: easyfix
> Fix For: 1.10.2
>
>
> BigQueryOperator currently sets default value of udf_config to False. This no
> longer works due to [https://github.com/apache/incubator-airflow/pull/3733]
> validating the type of that parameter as either None or list. Default value
> needs to be changed to None.
> The line in question added in the commit referenced above
> {code:java}
> (udf_config, 'userDefinedFunctionResources', None, list),
> {code}
>
>
> Note, other users of the hook may potentially encounter the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3479) Records of "Log" table should be kept (by default) when users delete a DAG.
[ https://issues.apache.org/jira/browse/AIRFLOW-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713458#comment-16713458 ] Kaxil Naik commented on AIRFLOW-3479: - Resolved by https://github.com/apache/incubator-airflow/pull/4287 > Records of "Log" table should be kept (by default) when users delete a DAG. > --- > > Key: AIRFLOW-3479 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3479 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.2 > > > Currently when we delete a DAG (using API or from the UI), it will delete all > related records in all tables (all tables in which "dag_id" is available), > including "log" table. > However, the records in "log" table should be kept (by default). This would > be ideal for multiple reasons, like auditing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3479) Records of "Log" table should be kept (by default) when users delete a DAG.
[ https://issues.apache.org/jira/browse/AIRFLOW-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3479. - Resolution: Fixed Fix Version/s: 1.10.2 > Records of "Log" table should be kept (by default) when users delete a DAG. > --- > > Key: AIRFLOW-3479 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3479 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.2 > > > Currently when we delete a DAG (using API or from the UI), it will delete all > related records in all tables (all tables in which "dag_id" is available), > including "log" table. > However, the records in "log" table should be kept (by default). This would > be ideal for multiple reasons, like auditing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3484) The KubernetesExecutor is overly verbose and this can kill schedulers.
[ https://issues.apache.org/jira/browse/AIRFLOW-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3484. - Resolution: Fixed Fix Version/s: 2.0.0 Resolved by https://github.com/apache/incubator-airflow/pull/4296 > The KubernetesExecutor is overly verbose and this can kill schedulers. > -- > > Key: AIRFLOW-3484 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3484 > Project: Apache Airflow > Issue Type: Bug > Components: kubernetes >Affects Versions: 1.10.1 >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Major > Fix For: 2.0.0 > > > There are two log lines in the k8sexecutor that can cause schedulers to crash > just by their sheer verbosity. This PR will switch these lines to debug as to > not mess with normal workflows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3408) Systemd setup instructions mention deprecated variable
[
https://issues.apache.org/jira/browse/AIRFLOW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3408.
-
Resolution: Fixed
> Systemd setup instructions mention deprecated variable
> --
>
> Key: AIRFLOW-3408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3408
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Victor Villas Bôas Chaves
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.2
>
>
> AIRFLOW-1698 was solved in code, but the documentation drifted.
> Places where {{SCHEDULER_RUNS}} is still mentioned as necessary adjustment:
> [https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/scripts/systemd/README]
> [https://github.com/apache/incubator-airflow/blob/b9fc03ea1ad5cea3c3aa668fcaca103f84167b9c/docs/howto/run-with-systemd.rst]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3406) Implement an Azure CosmosDB operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3406. - Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4265 > Implement an Azure CosmosDB operator > - > > Key: AIRFLOW-3406 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3406 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Tom Miller >Assignee: Tom Miller >Priority: Major > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik reassigned AIRFLOW-3447:
---
Assignee: Kaxil Naik
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Assignee: Kaxil Naik
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3447:
Fix Version/s: 1.10.2
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3447) Intended usage of ts_nodash macro broken with migration to new time system.
[
https://issues.apache.org/jira/browse/AIRFLOW-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710606#comment-16710606
]
Kaxil Naik commented on AIRFLOW-3447:
-
[~ashb] Timezone is giving few headaches isn't it. For 1.10.2 can we just
resolve this with this solution given above? If there are many issues.
> Intended usage of ts_nodash macro broken with migration to new time system.
> ---
>
> Key: AIRFLOW-3447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3447
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Reporter: Luka Draksler
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.2
>
>
> Migration to timezone aware times broke the intended usage of ts_nodash macro.
> ts_nodash is used in certain placeholders to create different names (table
> names, cluster names...). As such it is alphanumeric only, it contains no
> characters that could be deemed illegal by various naming restrictions.
> Migration to new time system changed that.
> As an example, this would be returned currently:
> {{20181205T125657.169324+}}
> {{before:}}
> {{20181204T03}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2440: Priority: Trivial (was: Major) > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Trivial > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2440: Affects Version/s: (was: 2.0.0) > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Trivial > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2440) Google Cloud SQL import/export operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2440. - Resolution: Fixed Fix Version/s: 1.10.2 > Google Cloud SQL import/export operator > --- > > Key: AIRFLOW-2440 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2440 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Guillermo Rodríguez Cano >Assignee: Szymon Przedwojski >Priority: Major > Fix For: 1.10.2 > > > Implement Google Cloud SQL Operator for import/export operations as a start. > Such operator implies the corresponding hook, which can be inspired by the > Google Dataflow hook for the management of the import/export jobs (checking > the status of an issued job). > API for [mysql|https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/] > and [postgres|https://cloud.google.com/sql/docs/postgres/admin-api/v1beta4/] > (they seem alike for these two operations at least) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1552) Airflow Filter_by_owner not working with password_auth
[ https://issues.apache.org/jira/browse/AIRFLOW-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-1552: Fix Version/s: 1.10.2 > Airflow Filter_by_owner not working with password_auth > -- > > Key: AIRFLOW-1552 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1552 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.8.0 > Environment: CentOS , python 2.7 >Reporter: raghu ram reddy >Priority: Major > Fix For: 1.10.2 > > > Airflow Filter_by_owner parameter is not working with password_auth. > I created sample user using the below code from airflow documentation and > enabled password_auth. > I'm able to login as the user created but by default this user is superuser > and there is noway to modify it, default all users created by PasswordUser > are superusers. > import airflow > from airflow import models, settings > from airflow.contrib.auth.backends.password_auth import PasswordUser > user = PasswordUser(models.User()) > user.username = 'test1' > user.password = 'test1' > user.is_superuser() > session = settings.Session() > session.add(user) > session.commit() > session.close() > exit() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2670) SSHOperator's timeout parameter doesn't affect SSHook timeoot
[ https://issues.apache.org/jira/browse/AIRFLOW-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2670: Fix Version/s: (was: 2.0.0) 1.10.2 > SSHOperator's timeout parameter doesn't affect SSHook timeoot > - > > Key: AIRFLOW-2670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2670 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 2.0.0 >Reporter: jin zhang >Priority: Major > Fix For: 1.10.2 > > > when I use SSHOperator, SSHOperator's timeout parameter can't set in SSHHook > and it's just effect exce_command. > old version: > self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id) > I change it to : > self.ssh_hook = SSHHook(ssh_conn_id=self.ssh_conn_id, timeout=self.timeout) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3309) Missing Mongo DB connection type
[ https://issues.apache.org/jira/browse/AIRFLOW-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3309: Fix Version/s: (was: 2.0.0) 1.10.2 > Missing Mongo DB connection type > > > Key: AIRFLOW-3309 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3309 > Project: Apache Airflow > Issue Type: Bug > Components: database >Affects Versions: 1.10.0 >Reporter: John Cheng >Assignee: John Cheng >Priority: Minor > Fix For: 1.10.2 > > > Unable to choose Mongo DB on the admin console connection page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-3438) BigQueryOperator should default udf_config to None instead of false.
[
https://issues.apache.org/jira/browse/AIRFLOW-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-3438 started by Kaxil Naik.
---
> BigQueryOperator should default udf_config to None instead of false.
>
>
> Key: AIRFLOW-3438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3438
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Reporter: Luka Draksler
>Assignee: Kaxil Naik
>Priority: Major
> Labels: easyfix
> Fix For: 1.10.2
>
>
> BigQueryOperator currently sets default value of udf_config to False. This no
> longer works due to [https://github.com/apache/incubator-airflow/pull/3733]
> validating the type of that parameter as either None or list. Default value
> needs to be changed to None.
> The line in question added in the commit referenced above
> {code:java}
> (udf_config, 'userDefinedFunctionResources', None, list),
> {code}
>
>
> Note, other users of the hook may potentially encounter the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3438) BigQueryOperator should default udf_config to None instead of false.
[
https://issues.apache.org/jira/browse/AIRFLOW-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3438:
Fix Version/s: 1.10.2
> BigQueryOperator should default udf_config to None instead of false.
>
>
> Key: AIRFLOW-3438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3438
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Reporter: Luka Draksler
>Assignee: Kaxil Naik
>Priority: Major
> Labels: easyfix
> Fix For: 1.10.2
>
>
> BigQueryOperator currently sets default value of udf_config to False. This no
> longer works due to [https://github.com/apache/incubator-airflow/pull/3733]
> validating the type of that parameter as either None or list. Default value
> needs to be changed to None.
> The line in question added in the commit referenced above
> {code:java}
> (udf_config, 'userDefinedFunctionResources', None, list),
> {code}
>
>
> Note, other users of the hook may potentially encounter the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3434) SFTPOperator does not create intermediate directories
[ https://issues.apache.org/jira/browse/AIRFLOW-3434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3434. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/4270 > SFTPOperator does not create intermediate directories > - > > Key: AIRFLOW-3434 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3434 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.2 > > > When using SFTPOperator with either 'get' or 'put', it doesn't create the > intermediate directories when copying the file and fails with directory does > not exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3434) SFTPOperator does not create intermediate directories
Kaxil Naik created AIRFLOW-3434: --- Summary: SFTPOperator does not create intermediate directories Key: AIRFLOW-3434 URL: https://issues.apache.org/jira/browse/AIRFLOW-3434 Project: Apache Airflow Issue Type: New Feature Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 1.10.2 When using SFTPOperator with either 'get' or 'put', it doesn't create the intermediate directories when copying the file and fails with directory does not exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3414) reload_module not working with custom logging class
[ https://issues.apache.org/jira/browse/AIRFLOW-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3414: Affects Version/s: (was: 1.10.2) 1.10.1 > reload_module not working with custom logging class > --- > > Key: AIRFLOW-3414 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3414 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 1.10.2 > > > If using custom logging class, the reload_module in dag_processing.py will > fail because it will try to reload default logging class, which is not loaded > at the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3380) Metrics documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3380. - Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4219 > Metrics documentation > - > > Key: AIRFLOW-3380 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3380 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Bartosz Ługowski >Assignee: Bartosz Ługowski >Priority: Trivial > Fix For: 1.10.2 > > > At the moment there is no documentation about Airflow metrics: > * how to enable it > * which metrics are generated > * description of each metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3414) reload_module not working with custom logging class
[ https://issues.apache.org/jira/browse/AIRFLOW-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3414: Fix Version/s: 1.10.2 > reload_module not working with custom logging class > --- > > Key: AIRFLOW-3414 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3414 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 1.10.2 > > > If using custom logging class, the reload_module in dag_processing.py will > fail because it will try to reload default logging class, which is not loaded > at the first place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3408) Systemd setup instructions mention deprecated variable
[
https://issues.apache.org/jira/browse/AIRFLOW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3408:
Fix Version/s: 1.10.2
> Systemd setup instructions mention deprecated variable
> --
>
> Key: AIRFLOW-3408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3408
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Victor Villas Bôas Chaves
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.2
>
>
> AIRFLOW-1698 was solved in code, but the documentation drifted.
> Places where {{SCHEDULER_RUNS}} is still mentioned as necessary adjustment:
> [https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/scripts/systemd/README]
> [https://github.com/apache/incubator-airflow/blob/b9fc03ea1ad5cea3c3aa668fcaca103f84167b9c/docs/howto/run-with-systemd.rst]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-3408) Systemd setup instructions mention deprecated variable
[
https://issues.apache.org/jira/browse/AIRFLOW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-3408 started by Kaxil Naik.
---
> Systemd setup instructions mention deprecated variable
> --
>
> Key: AIRFLOW-3408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3408
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Victor Villas Bôas Chaves
>Assignee: Kaxil Naik
>Priority: Minor
>
> AIRFLOW-1698 was solved in code, but the documentation drifted.
> Places where {{SCHEDULER_RUNS}} is still mentioned as necessary adjustment:
> [https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/scripts/systemd/README]
> [https://github.com/apache/incubator-airflow/blob/b9fc03ea1ad5cea3c3aa668fcaca103f84167b9c/docs/howto/run-with-systemd.rst]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3408) Systemd setup instructions mention deprecated variable
[
https://issues.apache.org/jira/browse/AIRFLOW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik reassigned AIRFLOW-3408:
---
Assignee: Kaxil Naik (was: Anoop Kunjuraman)
> Systemd setup instructions mention deprecated variable
> --
>
> Key: AIRFLOW-3408
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3408
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Victor Villas Bôas Chaves
>Assignee: Kaxil Naik
>Priority: Minor
>
> AIRFLOW-1698 was solved in code, but the documentation drifted.
> Places where {{SCHEDULER_RUNS}} is still mentioned as necessary adjustment:
> [https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/scripts/systemd/README]
> [https://github.com/apache/incubator-airflow/blob/b9fc03ea1ad5cea3c3aa668fcaca103f84167b9c/docs/howto/run-with-systemd.rst]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3432) Test is missing for feature "Delete DAG in UI"
[ https://issues.apache.org/jira/browse/AIRFLOW-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3432. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/4266 > Test is missing for feature "Delete DAG in UI" > -- > > Key: AIRFLOW-3432 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3432 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > Fix For: 1.10.2 > > > In [https://github.com/apache/incubator-airflow/pull/4069,] "delete DAG in > UI" feature is fixed, but no test was added for it. > > We should have test for it to release it with more confidence. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3266) AWS Athena Operator in Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3266: Fix Version/s: (was: 2.0.0) 1.10.2 > AWS Athena Operator in Airflow > -- > > Key: AIRFLOW-3266 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3266 > Project: Apache Airflow > Issue Type: New Feature > Components: aws >Affects Versions: 1.10.0 >Reporter: Sai Phanindhra >Assignee: Sai Phanindhra >Priority: Minor > Fix For: 1.10.2 > > > There is no official athena operator as of now airflow. Either one has do it > using boto3 in python operator or using aws cli in bash operator. Either of > these does not take care of total life cycle of query. Create a Athena > operator and hook to submit presto query and update task based of state of > submitted query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3022) Add volume mount to KubernetesExecutorConfig
[ https://issues.apache.org/jira/browse/AIRFLOW-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3022. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/3855 > Add volume mount to KubernetesExecutorConfig > > > Key: AIRFLOW-3022 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3022 > Project: Apache Airflow > Issue Type: Improvement > Components: executor >Affects Versions: 1.10.0 >Reporter: John Cheng >Assignee: John Cheng >Priority: Minor > Fix For: 1.10.2 > > > Allows volume mount to the worker pod with executor_config. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3361) Add the task_id to the Deprecation Warning when passing unsupported keywords to BaseOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3361:
Fix Version/s: (was: 2.0.0)
> Add the task_id to the Deprecation Warning when passing unsupported keywords
> to BaseOperator
>
>
> Key: AIRFLOW-3361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3361
> Project: Apache Airflow
> Issue Type: Improvement
> Components: logging
>Affects Versions: 1.9.0
>Reporter: Martin Black
>Assignee: Martin Black
>Priority: Trivial
> Fix For: 1.10.2
>
>
> In 2.0 passing invalid keywords to {{BaseOperator}} will be deprecated. Prior
> to that, there is a {{PendingDeprecationWarning}} raised, however it can be
> hard to track down which specific task is raising this warning.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3425) GoogleCloudBaseHook doesn't set default scope when google_cloud_default connection is empty
[
https://issues.apache.org/jira/browse/AIRFLOW-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3425:
Fix Version/s: (was: 2.0.0)
1.10.2
> GoogleCloudBaseHook doesn't set default scope when google_cloud_default
> connection is empty
> ---
>
> Key: AIRFLOW-3425
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3425
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp, hooks
>Affects Versions: 1.10.1
>Reporter: Alexander Sokolov
>Priority: Minor
> Fix For: 1.10.2
>
>
> When using Application Default Credentials to authenticating to GCP.
> _If the environment variable {{GOOGLE_APPLICATION_CREDENTIALS}} is set to the
> path of a valid service account JSON private key file, then it is loaded and
> returned. The project ID returned is the project ID defined in the service
> account file if available (some older files do not contain project ID
> information)._
> {code:java}
> [2018-11-30 11:18:35,038] {models.py:1569} INFO - Executing
> on
> 2018-11-29T17:54:03.110575+00:00
> [2018-11-30 11:18:35,039] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', 'airflow run transform_raw_data upload_transformed_data_to_gcs
> 2018-11-29T17:54:03.110575+00:00 --job_id 98 --raw -sd
> DAGS_FOLDER/transform_raw_data.py --cfg_path
> /var/folders/5j/n3y23gt93l19lmt1tf4mgpr8gn/T/tmprbpmva2a']
> [2018-11-30 11:18:35,606] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,606] {settings.py:174}
> INFO - setting.configure_orm(): Using pool settings. pool_size=5,
> pool_recycle=1800
> [2018-11-30 11:18:35,981] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,979] {_init_.py:51} INFO
> - Using executor LocalExecutor
> [2018-11-30 11:18:36,202] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:36,201] {models.py:258} INFO
> - Filling up the DagBag from /Users/alex/airflow/dags/transform_raw_data.py
> [2018-11-30 11:18:38,391] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:38,390] {base_hook.py:83}
> INFO - Using connection to: localhost
> [2018-11-30 11:18:43,427] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:43,426] {cli.py:492} INFO -
> Running 2018-11-29T17:54:03.110575+00:00 [running]> on host laptop.local
> [2018-11-30 11:18:48,574] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,574] {gcp_api_base_hook.py:86} INFO - Getting connection using
> `google.auth.default()` since no key file is defined for hook.
> [2018-11-30 11:18:48,583] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,583] {discovery.py:267} INFO - URL being requested: GET
> https://www.googleapis.com/discovery/v1/apis/storage/v1/rest
> [2018-11-30 11:18:48,852] {models.py:1736} ERROR - ('invalid_scope: Empty or
> missing scope not allowed.', '{\n "error": "invalid_scope",\n
> "error_description": "Empty or missing scope not allowed."\n}')
> Traceback (most recent call last):
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/operators/file_to_gcs.py",
> line 75, in execute
> filename=self.src)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 187, in upload
> service = self.get_conn()
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 48, in get_conn
> 'storage', 'v1', http=http_authorized, cache_discovery=False)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 222, in build
> requested_url, discovery_http, cache_discovery, cache)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 269, in _retrieve_discovery_doc
> resp, content = http.request(actual_url)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/google_auth_httplib2.py",
> line 187, in request
> self._request, method, uri,
[jira] [Updated] (AIRFLOW-3345) Add Google Cloud Storage (GCS) operators for ACL
[ https://issues.apache.org/jira/browse/AIRFLOW-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3345: Fix Version/s: (was: 2.0.0) > Add Google Cloud Storage (GCS) operators for ACL > > > Key: AIRFLOW-3345 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3345 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Reporter: Szymon Przedwojski >Assignee: Szymon Przedwojski >Priority: Minor > Fix For: 1.10.2 > > > Add operators to be able to add ACL entries to GCS buckets and objects: > * > [https://cloud.google.com/storage/docs/json_api/v1/objectAccessControls/insert] > * > [https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls/insert] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3213) Create ADLS to GCS operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3213: Fix Version/s: (was: 2.0.0) 1.10.2 > Create ADLS to GCS operator > > > Key: AIRFLOW-3213 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3213 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, operators >Reporter: Brandon Kvarda >Assignee: Brandon Kvarda >Priority: Minor > Fix For: 1.10.2 > > > Create ADLS to GCS operator that supports copying of files from ADLS to GCS -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3384) Allow higher versions of sqlalchemy and jinja
[ https://issues.apache.org/jira/browse/AIRFLOW-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3384: Fix Version/s: (was: 2.0.0) > Allow higher versions of sqlalchemy and jinja > - > > Key: AIRFLOW-3384 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3384 > Project: Apache Airflow > Issue Type: Improvement > Components: dependencies >Reporter: Jose Luis Ricon >Assignee: Jose Luis Ricon >Priority: Major > Fix For: 1.10.2 > > > At the moment airflow doesn't allow the installation of sqlalchemy version > 1.2.11 and jinja2==2.10 . Airflow works with both, and there is no reason to > allow higher versions. Projects downstream who are currently forcing the > installation of said versions, overriding airflow's dependencies, will > benefit for this as it will allow for version-compatible installations > without loss in functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2983) Add prev_ds_nodash and next_ds_nodash macro
[
https://issues.apache.org/jira/browse/AIRFLOW-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-2983.
-
Resolution: Fixed
Fix Version/s: 1.10.2
> Add prev_ds_nodash and next_ds_nodash macro
>
>
> Key: AIRFLOW-2983
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2983
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
> Fix For: 1.10.2
>
>
> Similar to [#3418|https://github.com/apache/incubator-airflow/pull/3418], but
> introduce two nodash macros:
> {\{ prev_ds_nodash }}: the previous execution date as \{{ MMDD }}
> {\{ next_ds_nodash }}: the next execution date as \{{ MMDD }}
> We introduced prev_ds and next_ds in
> [#3418|https://github.com/apache/incubator-airflow/pull/3418] which is
> beneficial for backfill. By using these two macros, we could override the
> schedule_internal from daily to certain days(e.g 7 days) in backfill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2983) Add prev_ds_nodash and next_ds_nodash macro
[
https://issues.apache.org/jira/browse/AIRFLOW-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik reopened AIRFLOW-2983:
-
> Add prev_ds_nodash and next_ds_nodash macro
>
>
> Key: AIRFLOW-2983
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2983
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Minor
> Fix For: 1.10.2
>
>
> Similar to [#3418|https://github.com/apache/incubator-airflow/pull/3418], but
> introduce two nodash macros:
> {\{ prev_ds_nodash }}: the previous execution date as \{{ MMDD }}
> {\{ next_ds_nodash }}: the next execution date as \{{ MMDD }}
> We introduced prev_ds and next_ds in
> [#3418|https://github.com/apache/incubator-airflow/pull/3418] which is
> beneficial for backfill. By using these two macros, we could override the
> schedule_internal from daily to certain days(e.g 7 days) in backfill.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3416) CloudSqlQueryOperator with sql proxy does not work with Python 3.x
[
https://issues.apache.org/jira/browse/AIRFLOW-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3416.
-
Resolution: Fixed
Resolved by https://github.com/apache/incubator-airflow/pull/4254
> CloudSqlQueryOperator with sql proxy does not work with Python 3.x
> --
>
> Key: AIRFLOW-3416
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3416
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.1
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.2
>
>
> There are compatibility issues with Python 3.x for CloudSQLoperator. Output
> of cloud_sql_proxy binary is parsed and the output in Python3 is bytes rather
> than string so several "in"s raise an exception in Python 3. It needs
> explicit decode('utf-8')
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3396) Redundant sql param check in BigQueryOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3396. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/4224 > Redundant sql param check in BigQueryOperator > - > > Key: AIRFLOW-3396 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3396 > Project: Apache Airflow > Issue Type: Task > Components: gcp >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Trivial > Fix For: 2.0.0 > > > The 'sql' parameter in BigQueryOperator accepts None and then there is a > check which checks if it is None and gives error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3403) Add AWS Athena Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3403. - Resolution: Fixed Fix Version/s: 1.10.2 Resolved by https://github.com/apache/incubator-airflow/pull/4244 > Add AWS Athena Sensor > - > > Key: AIRFLOW-3403 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3403 > Project: Apache Airflow > Issue Type: New Feature >Reporter: tal >Assignee: tal >Priority: Minor > Fix For: 1.10.2 > > > It will be nice to have an Athena sensor to monitor the progress of the query -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3403) Add AWS Athena Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3403: Summary: Add AWS Athena Sensor (was: create athena sensor) > Add AWS Athena Sensor > - > > Key: AIRFLOW-3403 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3403 > Project: Apache Airflow > Issue Type: Improvement >Reporter: tal >Assignee: tal >Priority: Minor > > It will be nice to have an Athena sensor to monitor the progress of the query -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3403) Add AWS Athena Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3403: Issue Type: New Feature (was: Improvement) > Add AWS Athena Sensor > - > > Key: AIRFLOW-3403 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3403 > Project: Apache Airflow > Issue Type: New Feature >Reporter: tal >Assignee: tal >Priority: Minor > > It will be nice to have an Athena sensor to monitor the progress of the query -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3323) Support Basic Authentication for Flower
[ https://issues.apache.org/jira/browse/AIRFLOW-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3323: Fix Version/s: (was: 2.0.0) 1.10.2 > Support Basic Authentication for Flower > --- > > Key: AIRFLOW-3323 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3323 > Project: Apache Airflow > Issue Type: New Feature > Components: celery >Affects Versions: 1.10.0, 1.10.1 >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 1.10.2 > > > The current `airflow flower` doesn't come with any authentication. This may > make essential information exposed to in an untrusted environment. > Currently Flower itself supports > * HTTP Basic Authentication > * Google OAuth 2.0 > * GitHub OAuth > Given Flower is not really the most essential component of Airflow, we don't > have to support all its authentication methods. But may be good to at least > support Basic Authentication. > > This ticket adds support to Basic Authentication for `Airflow Flower`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3425) GoogleCloudBaseHook doesn't set default scope when google_cloud_default connection is empty
[
https://issues.apache.org/jira/browse/AIRFLOW-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705951#comment-16705951
]
Kaxil Naik commented on AIRFLOW-3425:
-
Btw [~alsokoloff] it is not a bug, it already checked for `is not None`, and
not `is None` as you mentioned.
However, I am merging this because it is a better convention.
> GoogleCloudBaseHook doesn't set default scope when google_cloud_default
> connection is empty
> ---
>
> Key: AIRFLOW-3425
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3425
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp, hooks
>Affects Versions: 1.10.1
>Reporter: Alexander Sokolov
>Priority: Minor
> Fix For: 2.0.0
>
>
> When using Application Default Credentials to authenticating to GCP.
> _If the environment variable {{GOOGLE_APPLICATION_CREDENTIALS}} is set to the
> path of a valid service account JSON private key file, then it is loaded and
> returned. The project ID returned is the project ID defined in the service
> account file if available (some older files do not contain project ID
> information)._
> {code:java}
> [2018-11-30 11:18:35,038] {models.py:1569} INFO - Executing
> on
> 2018-11-29T17:54:03.110575+00:00
> [2018-11-30 11:18:35,039] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', 'airflow run transform_raw_data upload_transformed_data_to_gcs
> 2018-11-29T17:54:03.110575+00:00 --job_id 98 --raw -sd
> DAGS_FOLDER/transform_raw_data.py --cfg_path
> /var/folders/5j/n3y23gt93l19lmt1tf4mgpr8gn/T/tmprbpmva2a']
> [2018-11-30 11:18:35,606] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,606] {settings.py:174}
> INFO - setting.configure_orm(): Using pool settings. pool_size=5,
> pool_recycle=1800
> [2018-11-30 11:18:35,981] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,979] {_init_.py:51} INFO
> - Using executor LocalExecutor
> [2018-11-30 11:18:36,202] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:36,201] {models.py:258} INFO
> - Filling up the DagBag from /Users/alex/airflow/dags/transform_raw_data.py
> [2018-11-30 11:18:38,391] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:38,390] {base_hook.py:83}
> INFO - Using connection to: localhost
> [2018-11-30 11:18:43,427] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:43,426] {cli.py:492} INFO -
> Running 2018-11-29T17:54:03.110575+00:00 [running]> on host laptop.local
> [2018-11-30 11:18:48,574] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,574] {gcp_api_base_hook.py:86} INFO - Getting connection using
> `google.auth.default()` since no key file is defined for hook.
> [2018-11-30 11:18:48,583] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,583] {discovery.py:267} INFO - URL being requested: GET
> https://www.googleapis.com/discovery/v1/apis/storage/v1/rest
> [2018-11-30 11:18:48,852] {models.py:1736} ERROR - ('invalid_scope: Empty or
> missing scope not allowed.', '{\n "error": "invalid_scope",\n
> "error_description": "Empty or missing scope not allowed."\n}')
> Traceback (most recent call last):
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/operators/file_to_gcs.py",
> line 75, in execute
> filename=self.src)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 187, in upload
> service = self.get_conn()
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 48, in get_conn
> 'storage', 'v1', http=http_authorized, cache_discovery=False)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 222, in build
> requested_url, discovery_http, cache_discovery, cache)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 269, in _retrieve_discovery_doc
> resp, content = http.request(actual_url)
> File
>
[jira] [Resolved] (AIRFLOW-3425) GoogleCloudBaseHook doesn't set default scope when google_cloud_default connection is empty
[
https://issues.apache.org/jira/browse/AIRFLOW-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3425.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Resolved by https://github.com/apache/incubator-airflow/pull/4261
> GoogleCloudBaseHook doesn't set default scope when google_cloud_default
> connection is empty
> ---
>
> Key: AIRFLOW-3425
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3425
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp, hooks
>Affects Versions: 1.10.1
>Reporter: Alexander Sokolov
>Priority: Minor
> Fix For: 2.0.0
>
>
> When using Application Default Credentials to authenticating to GCP.
> _If the environment variable {{GOOGLE_APPLICATION_CREDENTIALS}} is set to the
> path of a valid service account JSON private key file, then it is loaded and
> returned. The project ID returned is the project ID defined in the service
> account file if available (some older files do not contain project ID
> information)._
> {code:java}
> [2018-11-30 11:18:35,038] {models.py:1569} INFO - Executing
> on
> 2018-11-29T17:54:03.110575+00:00
> [2018-11-30 11:18:35,039] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', 'airflow run transform_raw_data upload_transformed_data_to_gcs
> 2018-11-29T17:54:03.110575+00:00 --job_id 98 --raw -sd
> DAGS_FOLDER/transform_raw_data.py --cfg_path
> /var/folders/5j/n3y23gt93l19lmt1tf4mgpr8gn/T/tmprbpmva2a']
> [2018-11-30 11:18:35,606] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,606] {settings.py:174}
> INFO - setting.configure_orm(): Using pool settings. pool_size=5,
> pool_recycle=1800
> [2018-11-30 11:18:35,981] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:35,979] {_init_.py:51} INFO
> - Using executor LocalExecutor
> [2018-11-30 11:18:36,202] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:36,201] {models.py:258} INFO
> - Filling up the DagBag from /Users/alex/airflow/dags/transform_raw_data.py
> [2018-11-30 11:18:38,391] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:38,390] {base_hook.py:83}
> INFO - Using connection to: localhost
> [2018-11-30 11:18:43,427] {base_task_runner.py:107} INFO - Job 98: Subtask
> upload_transformed_data_to_gcs [2018-11-30 11:18:43,426] {cli.py:492} INFO -
> Running 2018-11-29T17:54:03.110575+00:00 [running]> on host laptop.local
> [2018-11-30 11:18:48,574] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,574] {gcp_api_base_hook.py:86} INFO - Getting connection using
> `google.auth.default()` since no key file is defined for hook.
> [2018-11-30 11:18:48,583] {logging_mixin.py:95} INFO - [2018-11-30
> 11:18:48,583] {discovery.py:267} INFO - URL being requested: GET
> https://www.googleapis.com/discovery/v1/apis/storage/v1/rest
> [2018-11-30 11:18:48,852] {models.py:1736} ERROR - ('invalid_scope: Empty or
> missing scope not allowed.', '{\n "error": "invalid_scope",\n
> "error_description": "Empty or missing scope not allowed."\n}')
> Traceback (most recent call last):
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/models.py",
> line 1633, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/operators/file_to_gcs.py",
> line 75, in execute
> filename=self.src)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 187, in upload
> service = self.get_conn()
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/airflow/contrib/hooks/gcs_hook.py",
> line 48, in get_conn
> 'storage', 'v1', http=http_authorized, cache_discovery=False)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/_helpers.py",
> line 130, in positional_wrapper
> return wrapped(*args, **kwargs)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 222, in build
> requested_url, discovery_http, cache_discovery, cache)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/googleapiclient/discovery.py",
> line 269, in _retrieve_discovery_doc
> resp, content = http.request(actual_url)
> File
> "/Users/alex/.local/share/virtualenvs/project-X9xQGmcz/lib/python3.6/site-packages/google_auth_httplib2.py",
> line
[jira] [Updated] (AIRFLOW-3392) Add index on dag_id in sla_miss table
[ https://issues.apache.org/jira/browse/AIRFLOW-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3392: Fix Version/s: (was: 1.10.2) 2.0.0 > Add index on dag_id in sla_miss table > - > > Key: AIRFLOW-3392 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3392 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 2.0.0 > > > The select queries on sla_miss table produce a great % of DB traffic and thus > made the DB CPU usage unnecessarily high. It would be a low hanging fruit to > add an index and reduce the load. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3062) Add Qubole in integration docs
[ https://issues.apache.org/jira/browse/AIRFLOW-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3062: Fix Version/s: (was: 2.0.0) 1.10.2 > Add Qubole in integration docs > -- > > Key: AIRFLOW-3062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3062 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2596) Oracle to Azure Datalake Transfer Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2596: Fix Version/s: (was: 2.0.0) 1.10.2 > Oracle to Azure Datalake Transfer Operator > --- > > Key: AIRFLOW-2596 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2596 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Marcus Rehm >Assignee: Marcus Rehm >Priority: Trivial > Fix For: 1.10.2 > > > Like MySQLToHveTransfer operator would be nice to have a > OracleToAzureDataLakeTransfer operator so we can simplify data extraction > from Oracle database and put it right into Azure Datalake. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2795) Oracle to Oracle Transfer Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2795: Fix Version/s: (was: 2.0.0) > Oracle to Oracle Transfer Operator > --- > > Key: AIRFLOW-2795 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2795 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Marcus Rehm >Assignee: Marcus Rehm >Priority: Trivial > Fix For: 1.10.2 > > > This operator should help in transfer data from one Oracle instance to > another or between tables in the same instance. t's suitable in use cases > where you don't want to or it's not allowed use dblink. > The operator needs a sql query and a destination table in order to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2753) dataproc_operator job operators provide a way to get the actual jobId submitted to Dataproc
[
https://issues.apache.org/jira/browse/AIRFLOW-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-2753:
Fix Version/s: 1.10.2
> dataproc_operator job operators provide a way to get the actual jobId
> submitted to Dataproc
> ---
>
> Key: AIRFLOW-2753
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2753
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, operators
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Minor
> Labels: dataproc, google, google-compute-engine, operator
> Fix For: 1.10.2, 2.0.0
>
>
> I'd like to be able to get the {{jobId}} that's submitted by the
> {{DataProcHook}} in order to do things like build the URL to the Dataproc job
> UI, etc. The {{job_name}} that the caller passes into
> {{DataProcPySparkOperator}}, etc gets an 8 character random string appended
> to it before the job is submitted. I don't currently see a way to access
> that value, which is needed for the above mentioned use case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2795) Oracle to Oracle Transfer Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2795: Fix Version/s: 1.10.2 > Oracle to Oracle Transfer Operator > --- > > Key: AIRFLOW-2795 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2795 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Marcus Rehm >Assignee: Marcus Rehm >Priority: Trivial > Fix For: 1.10.2 > > > This operator should help in transfer data from one Oracle instance to > another or between tables in the same instance. t's suitable in use cases > where you don't want to or it's not allowed use dblink. > The operator needs a sql query and a destination table in order to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2705) Move class-level moto decorator to method-level
[
https://issues.apache.org/jira/browse/AIRFLOW-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-2705:
Fix Version/s: (was: 2.0.0)
1.10.2
> Move class-level moto decorator to method-level
> ---
>
> Key: AIRFLOW-2705
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2705
> Project: Apache Airflow
> Issue Type: Bug
> Components: redshift, tests
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 1.10.2
>
>
> In tests/contrib/hooks/test_redshift_hook.py and
> tests/contrib/sensors/test_aws_redshift_cluster_sensor.py, moto decorators
> are at class-level, whereas they are at method-level in other tests using
> moto.
> {code:title=tests/contrib/hooks/test_redshift_hook.py}
> @mock_redshift
> class TestRedshiftHook(unittest.TestCase):
> {code}
> I found it can affect other tests using HTTP request. Let's say the following
> test is added to tests/hooks/test_http_hook.py:
> {code:title=tests/hooks/test_http_hook.py}
> def test_sample(self):
> HttpHook("GET").run("")
> {code}
> This test sends a GET request to https://www.google.com/ and succeeds of
> course:
> {code}
> $ ./run_unit_tests.sh tests.hooks.test_http_hook:TestHttpHook.test_sample
> (snip)
> /home/sekikn/a3/lib/python3.5/site-packages/urllib3/connectionpool.py:858:
> InsecureRequestWarning: Unverified HTTPS request is being made. Adding
> certificate verification is strongly advised. See:
> https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
> InsecureRequestWarning)
> .
> --
> Ran 1 test in 0.189s
> OK
> {code}
> But strangely, it fails with ConnectionError when it's executed after
> tests/contrib/hooks/test_redshift_hook.py or
> tests/contrib/sensors/test_aws_redshift_cluster_sensor.py:
> {code}
> $ ./run_unit_tests.sh tests.contrib.hooks.test_redshift_hook
> tests.hooks.test_http_hook:TestHttpHook.test_sample
> (snip)
> E
> ==
> ERROR: test_sample (tests.hooks.test_http_hook.TestHttpHook)
> --
> Traceback (most recent call last):
> File "/home/sekikn/dev/incubator-airflow/tests/hooks/test_http_hook.py",
> line 250, in test_sample
> HttpHook("GET").run("")
> File "/home/sekikn/dev/incubator-airflow/airflow/hooks/http_hook.py", line
> 119, in run
> return self.run_and_check(session, prepped_request, extra_options)
> File "/home/sekikn/dev/incubator-airflow/airflow/hooks/http_hook.py", line
> 167, in run_and_check
> raise ex
> File "/home/sekikn/dev/incubator-airflow/airflow/hooks/http_hook.py", line
> 159, in run_and_check
> allow_redirects=extra_options.get("allow_redirects", True))
> File "/home/sekikn/a3/lib/python3.5/site-packages/requests/sessions.py",
> line 618, in send
> r = adapter.send(request, **kwargs)
> File
> "/home/sekikn/a3/lib/python3.5/site-packages/moto/packages/responses/responses.py",
> line 308, in unbound_on_send
> return self._on_request(adapter, request, *a, **kwargs)
> File
> "/home/sekikn/a3/lib/python3.5/site-packages/moto/packages/responses/responses.py",
> line 250, in _on_request
> raise response
> requests.exceptions.ConnectionError: Connection refused: GET
> https://www.google.com/
> >> begin captured stdout << -
> [2018-06-30 09:41:21,029] {base_hook.py:83} INFO - Using connection to:
> https://www.google.com/
> [2018-06-30 09:41:21,030] {http_hook.py:118} INFO - Sending 'GET' to url:
> https://www.google.com/
> [2018-06-30 09:41:21,030] {http_hook.py:166} WARNING - Connection refused:
> GET https://www.google.com/ Tenacity will retry to execute the operation
> - >> end captured stdout << --
> >> begin captured logging <<
> airflow.utils.log.logging_mixin.LoggingMixin: INFO: Using connection to:
> https://www.google.com/
> airflow.hooks.http_hook.HttpHook: INFO: Sending 'GET' to url:
> https://www.google.com/
> airflow.hooks.http_hook.HttpHook: WARNING: Connection refused: GET
> https://www.google.com/ Tenacity will retry to execute the operation
> - >> end captured logging << -
> --
> Ran 7 tests in 1.838s
> FAILED (errors=1)
> {code}
> This problem is solved by using method-level moto decorator instead of
> class-level one. I suspect class-level decorator patches method that sends
> HTTP request and doesn't restore the original state.
--
This message was sent by Atlassian JIRA
[jira] [Updated] (AIRFLOW-3365) Missing visibility_timeout will overwrite celery broker transport options
[
https://issues.apache.org/jira/browse/AIRFLOW-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3365:
Fix Version/s: (was: 2.0.0)
1.10.2
> Missing visibility_timeout will overwrite celery broker transport options
> -
>
> Key: AIRFLOW-3365
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3365
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.10.0
>Reporter: Victor Villas Bôas Chaves
>Assignee: Victor Villas Bôas Chaves
>Priority: Major
> Fix For: 1.10.2
>
>
> If you specify the region in your airflow.cfg like this:
> {{[celery_broker_transport_options] }}
> {{region = us-east-2 }}
> it gets ignored.
> This is ignored because Airflow checks to see if the {{visibility_timeout}}
> parameter is set in the transport options and, if it's not set, replaces the
> transport options completely with {{{'visibility_timeout': 21600}}} (at
> [https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_celery.py#L40]).
> If you add {{visibility_timeout}}, the region is correctly set:
> {{[celery_broker_transport_options] }}
> {{region = us-east-2 }}
> {{visibility_timeout = 21600 }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3294) Add QuboleCheckOperator in integration docs
[ https://issues.apache.org/jira/browse/AIRFLOW-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3294: Fix Version/s: (was: 2.0.0) 1.10.2 > Add QuboleCheckOperator in integration docs > --- > > Key: AIRFLOW-3294 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3294 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2867) Airflow Python Code not compatible to coding guidelines and standards
[ https://issues.apache.org/jira/browse/AIRFLOW-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2867: Fix Version/s: (was: 2.0.0) 1.10.2 > Airflow Python Code not compatible to coding guidelines and standards > -- > > Key: AIRFLOW-2867 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2867 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.2 > > > Some of the Airflow code doesn't conform to python coding guidelines and > standards. > The improvement I have analyzed are below: > - Dictionary creation should be written by dictionary literal > - Mutable default argument. Python’s default arguments are evaluated once > when the function is defined, not each time the function is called (like it > is in say, Ruby). This means that if you use a mutable default argument and > mutate it, you will and have mutated that object for all future calls to the > function as well. > - Functions calling sets can be replaced by set literal > - Replace list literals > - Some of the static methods haven't been set static > - Redundant parentheses -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3343) DockerOperator always reports containers as failed
[ https://issues.apache.org/jira/browse/AIRFLOW-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3343: Fix Version/s: (was: 1.10.1) 1.10.2 > DockerOperator always reports containers as failed > -- > > Key: AIRFLOW-3343 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3343 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.1 >Reporter: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.2 > > > The docker-py 3.0.0 has some API breaking changes that didn't get picked up > because our mocking was still returning the old API. From > https://docker-py.readthedocs.io/en/stable/change-log.html#breaking-changes: > > APIClient.wait and Container.wait now return a dict representing the API’s > > response instead of returning the status code directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3392) Add index on dag_id in sla_miss table
[ https://issues.apache.org/jira/browse/AIRFLOW-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3392. - Resolution: Fixed Fix Version/s: 1.10.2 > Add index on dag_id in sla_miss table > - > > Key: AIRFLOW-3392 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3392 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 1.10.2 > > > The select queries on sla_miss table produce a great % of DB traffic and thus > made the DB CPU usage unnecessarily high. It would be a low hanging fruit to > add an index and reduce the load. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3301) Update CI test for [AIRFLOW-3132] (PR #3977)
[ https://issues.apache.org/jira/browse/AIRFLOW-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3301: Fix Version/s: 1.10.2 > Update CI test for [AIRFLOW-3132] (PR #3977) > > > Key: AIRFLOW-3301 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3301 > Project: Apache Airflow > Issue Type: Test > Components: tests >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > Fix For: 1.10.2 > > > In PR [https://github.com/apache/incubator-airflow/pull/3977,] test is not > updated accordingly, and it results in CI failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2691) Make Airflow's JS code (and dependencies) manageable via npm and webpack
[ https://issues.apache.org/jira/browse/AIRFLOW-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2691: Fix Version/s: (was: 2.0.0) 1.10.2 > Make Airflow's JS code (and dependencies) manageable via npm and webpack > > > Key: AIRFLOW-2691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2691 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Major > Fix For: 1.10.2 > > > Airflow's JS code is hard to maintain and upgrade. The dependencies are > locally existing files making it hard to upgrade versions. > Make sure Airflow uses *npm* and *webpack* for the dependencies management. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3236) Create AzureDataLakeStorageListOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3236: Fix Version/s: (was: 2.0.0) 1.10.2 > Create AzureDataLakeStorageListOperator > --- > > Key: AIRFLOW-3236 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3236 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Reporter: Brandon Kvarda >Assignee: Brandon Kvarda >Priority: Minor > Fix For: 1.10.2 > > > Creates an Operator that is similar to GoogleCloudStorageListOperator and > S3ListOperator that returns a list of files at some specified path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3395) add to documentation all existed REST API endpoints and example how to pass dag_runs params
[ https://issues.apache.org/jira/browse/AIRFLOW-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3395: Fix Version/s: (was: 2.0.0) 1.10.2 > add to documentation all existed REST API endpoints and example how to pass > dag_runs params > --- > > Key: AIRFLOW-3395 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3395 > Project: Apache Airflow > Issue Type: Task >Affects Versions: 1.10.2 >Reporter: Iuliia Volkova >Assignee: Iuliia Volkova >Priority: Minor > Fix For: 1.10.2 > > > In doc exist only 2 endpoints https://airflow.apache.org/api.html#endpoints > In source code > (https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/www_rbac/api/experimental/endpoints.py) > we have more, caused issues when users think that there are not more methods > - I got on work project several questions about it and also saw in > stackoverflow relative questions: > https://stackoverflow.com/questions/50121593/pass-parameters-to-airflow-experimental-rest-api-when-creating-dag-run > > I want to add more information about REST API -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2642) [kubernetes executor worker] the value of git-sync init container ENV GIT_SYNC_ROOT is wrong
[
https://issues.apache.org/jira/browse/AIRFLOW-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-2642:
Fix Version/s: (was: 2.0.0)
1.10.2
> [kubernetes executor worker] the value of git-sync init container ENV
> GIT_SYNC_ROOT is wrong
>
>
> Key: AIRFLOW-2642
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2642
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.0
>Reporter: pengchen
>Assignee: pengchen
>Priority: Major
> Fix For: 1.10.2
>
>
> There are two way of syncing dags, pvc and git-sync. When we use git-sync
> this way, the generated worker pod yaml file fragment is as follows
>
> {code:java}
> worker container:
> ---
> containers:
> - args:
> - airflow run tutorial1 print_date 2018-06-19T07:57:15.011693+00:00 --local
> -sd
> /root/airflow/dags/dags/example_dags/tutorial1.py
> command:
> - bash
> - -cx
> - --
> env:
> - name: AIRFLOW__CORE__AIRFLOW_HOME
> value: /root/airflow
> - name: AIRFLOW__CORE__EXECUTOR
> value: LocalExecutor
> - name: AIRFLOW__CORE__DAGS_FOLDER
> value: /tmp/dags
> - name: SQL_ALCHEMY_CONN
> valueFrom:
> secretKeyRef:
> key: sql_alchemy_conn
> name: airflow-secrets
> init container:
> ---
> initContainers:
> - env:
> - name: GIT_SYNC_REPO
> value: https://code.devops.xiaohongshu.com/pengchen/Airflow-DAGs.git
> - name: GIT_SYNC_BRANCH
> value: master
> - name: GIT_SYNC_ROOT
> value: /tmp
> - name: GIT_SYNC_DEST
> value: dags
> - name: GIT_SYNC_ONE_TIME
> value: "true"
> - name: GIT_SYNC_USERNAME
> value: XXX
> - name: GIT_SYNC_PASSWORD
> value: XXX
> image: library/git-sync-amd64:v2.0.5
> imagePullPolicy: IfNotPresent
> name: git-sync-clone
> resources: {}
> securityContext:
> runAsUser: 0
> terminationMessagePath: /dev/termination-log
> terminationMessagePolicy: File
> volumeMounts:
> - mountPath: /root/airflow/dags/
> name: airflow-dags
> - mountPath: /root/airflow/logs
> name: airflow-logs
> - mountPath: /root/airflow/airflow.cfg
> name: airflow-config
> readOnly: true
> subPath: airflow.cfg
> - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
> name: default-token-xz87t
> readOnly: true
> {code}
> According to the configuration, git-sync will synchronize dags to /tmp/dags
> directory. However the worker container command args(airflow run tutorial1
> print_date 2018-06-19T07:57:15.011693+00:00 --local -sd
> /root/airflow/dags/dags/example_dags/tutorial1.py) are generated by the
> scheduler. Therefore, the task error is as follows
> {code:java}
> + airflow run tutorial1 print_date 2018-06-19T07:57:15.011693+00:00 --local
> -sd /root/airflow/dags/dags/example_dags/tutorial1.py
> [2018-06-19 07:57:29,075] {settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-06-19 07:57:29,232] {__init__.py:51} INFO - Using executor LocalExecutor
> [2018-06-19 07:57:29,373] {models.py:219} INFO - Filling up the DagBag from
> /root/airflow/dags/dags/example_dags/tutorial1.py
> [2018-06-19 07:57:29,648] {models.py:310} INFO - File
> /usr/local/lib/python2.7/dist-packages/airflow/example_dags/__init__.py
> assumed to contain no DAGs. Skipping.
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/cli.py", line 74,
> in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 475,
> in run
> dag = get_dag(args)
> File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 146,
> in get_dag
> 'parse.'.format(args.dag_id))
> airflow.exceptions.AirflowException: dag_id could not be found: tutorial1.
> Either the dag did not exist or it failed to parse.
> {code}
>
> The log shows that the worker cannot find the corresponding dag, so I think
> the environment variable GIT_SYNC_ROOT should be consistent with
> dag_volume_mount_path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3366) AirflowConfigParser getsection() doesn't pick environment overrides
[ https://issues.apache.org/jira/browse/AIRFLOW-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3366: Fix Version/s: (was: 2.0.0) 1.10.2 > AirflowConfigParser getsection() doesn't pick environment overrides > --- > > Key: AIRFLOW-3366 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3366 > Project: Apache Airflow > Issue Type: Bug > Components: configuration >Affects Versions: 1.10.0 >Reporter: Victor Villas Bôas Chaves >Assignee: Victor Villas Bôas Chaves >Priority: Major > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3353) redis-py 3.0.0 dependency breaks celery executor
[
https://issues.apache.org/jira/browse/AIRFLOW-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-3353:
Fix Version/s: (was: 2.0.0)
1.10.2
> redis-py 3.0.0 dependency breaks celery executor
>
>
> Key: AIRFLOW-3353
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3353
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Affects Versions: 1.10.0
>Reporter: Stefan Seelmann
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.2
>
>
> redis-py 3.0.0 was just released. Airflow 1.10.0 defines redis>=2.10.5 so
> installs redis-py 3.0.0 now.
> Error in worker below.
> Workaround: Pin redis==2.10.6 (e.g. in constraints.txt)
> {code}
> [2018-11-15 12:06:18,441: CRITICAL/MainProcess] Unrecoverable error:
> AttributeError("'float' object has no attribute 'items'",)
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/celery/worker/worker.py", line
> 205, in start
> self.blueprint.start(self)
> File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line
> 119, in start
> step.start(parent)
> File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line
> 369, in start
> return self.obj.start()
> File
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py",
> line 317, in start
> blueprint.start(self)
> File "/usr/local/lib/python3.6/site-packages/celery/bootsteps.py", line
> 119, in start
> step.start(parent)
> File
> "/usr/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py",
> line 593, in start
> c.loop(*c.loop_args())
> File "/usr/local/lib/python3.6/site-packages/celery/worker/loops.py", line
> 91, in asynloop
> next(loop)
> File "/usr/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py",
> line 354, in create_loop
> cb(*cbargs)
> File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py",
> line 1040, in on_readable
> self.cycle.on_readable(fileno)
> File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py",
> line 337, in on_readable
> chan.handlers[type]()
> File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py",
> line 724, in _brpop_read
> self.connection._deliver(loads(bytes_to_str(item)), dest)
> File
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py",
> line 983, in _deliver
> callback(message)
> File
> "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py",
> line 632, in _callback
> self.qos.append(message, message.delivery_tag)
> File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py",
> line 149, in append
> pipe.zadd(self.unacked_index_key, time(), delivery_tag) \
> File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2263,
> in zadd
> for pair in iteritems(mapping):
> File "/usr/local/lib/python3.6/site-packages/redis/_compat.py", line 123,
> in iteritems
> return iter(x.items())
> AttributeError: 'float' object has no attribute 'items'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3271) Airflow RBAC Permissions modification via UI do not persist
[ https://issues.apache.org/jira/browse/AIRFLOW-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3271: Fix Version/s: (was: 1.10.2) 2.0.0 > Airflow RBAC Permissions modification via UI do not persist > --- > > Key: AIRFLOW-3271 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3271 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.0 >Reporter: Smitha Koduri >Assignee: Smitha Koduri >Priority: Major > Fix For: 2.0.0 > > > After upgrading Airflow to 1.10, we have noticed that when attempting to add > a new permission-role mapping (via UI), initially it gets successfully added > to db. But later, the entry doesn't persist in the db. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2761) Parallelize Celery Executor enqueuing
[ https://issues.apache.org/jira/browse/AIRFLOW-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2761: Component/s: celery > Parallelize Celery Executor enqueuing > - > > Key: AIRFLOW-2761 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2761 > Project: Apache Airflow > Issue Type: Improvement > Components: celery >Reporter: Kevin Yang >Priority: Major > Fix For: 1.10.2, 2.0.0 > > > Currently celery executor enqueues in an async fashion but still doing that > in a single process loop. This can slows down scheduler loop and creates > scheduling delay if we have large # of task to schedule in a short time, e.g. > UTC midnight we need to schedule large # of sensors in a short period. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2399) TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime
[ https://issues.apache.org/jira/browse/AIRFLOW-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702446#comment-16702446 ] Kaxil Naik commented on AIRFLOW-2399: - Resolved by https://github.com/apache/incubator-airflow/pull/4228 > TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime > --- > > Key: AIRFLOW-2399 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2399 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: andy dreyfuss >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > Currently TriggerDagRunOperator takes a string argument to specify the one > dag id that is triggered on success. But it would be just as easy and more > flexible to allow any. Incoming PR to demo what I mean ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2399) TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime
[ https://issues.apache.org/jira/browse/AIRFLOW-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2399: External issue URL: (was: https://github.com/apache/incubator-airflow/pull/4228) > TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime > --- > > Key: AIRFLOW-2399 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2399 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: andy dreyfuss >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > Currently TriggerDagRunOperator takes a string argument to specify the one > dag id that is triggered on success. But it would be just as easy and more > flexible to allow any. Incoming PR to demo what I mean ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2399) TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime
[ https://issues.apache.org/jira/browse/AIRFLOW-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2399: External issue URL: https://github.com/apache/incubator-airflow/pull/4228 > TriggerDagRunOperator Should Permit Implementor to Choose Dag Id at Runtime > --- > > Key: AIRFLOW-2399 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2399 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Reporter: andy dreyfuss >Assignee: Kaxil Naik >Priority: Minor > Fix For: 2.0.0 > > > Currently TriggerDagRunOperator takes a string argument to specify the one > dag id that is triggered on success. But it would be just as easy and more > flexible to allow any. Incoming PR to demo what I mean ... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3410) Remote Host Identification Has Changed Error while using SSHOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3410. - Resolution: Fixed Resolved by https://github.com/apache/incubator-airflow/pull/4249 > Remote Host Identification Has Changed Error while using SSHOperator > > > Key: AIRFLOW-3410 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3410 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.2 > > > Currently, there is no provision in Airflow's implementation of SSHOperator > to disable Host Checking and use a Null Known Host file to allow connecting > to host that has a new IP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3346) Add gcp transfer service hook and operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3346: Issue Type: New Feature (was: Bug) > Add gcp transfer service hook and operators > --- > > Key: AIRFLOW-3346 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3346 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Josh Carp >Priority: Minor > Fix For: 1.10.2 > > > Add a hook and operator(s) to connect to gcp storage transfer service and > transfer files from s3 to gcp (and gcp to gcp) without copying to local disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3306) Disable unused flask-sqlalchemy modification tracking
[ https://issues.apache.org/jira/browse/AIRFLOW-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3306: Issue Type: Improvement (was: Bug) > Disable unused flask-sqlalchemy modification tracking > - > > Key: AIRFLOW-3306 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3306 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Josh Carp >Assignee: Josh Carp >Priority: Trivial > Fix For: 1.10.2, 2.0.0 > > > By default, flask-sqlalchemy tracks model changes for its event system, which > adds some overhead. Since I don't think we're using the flask-sqlalchemy > event system, we should be able to turn off modification tracking and improve > performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2760) DAG parsing loop coupled with scheduler loop
[ https://issues.apache.org/jira/browse/AIRFLOW-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2760: Fix Version/s: (was: 2.0.0) 1.10.2 > DAG parsing loop coupled with scheduler loop > > > Key: AIRFLOW-2760 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2760 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 1.10.2 > > > Currently DAG parsing loop is coupled with scheduler loop, meaning that if > scheduler loop became slow, we will parse DAG slower. > As a simple producer and consumer pattern, we shall have them decoupled and > completely remove the scheduling bottleneck placed by DAG parsing--which is > identified in Airbnb as the current biggest bottleneck. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3410) Remote Host Identification Has Changed Error while using SSHOperator
Kaxil Naik created AIRFLOW-3410: --- Summary: Remote Host Identification Has Changed Error while using SSHOperator Key: AIRFLOW-3410 URL: https://issues.apache.org/jira/browse/AIRFLOW-3410 Project: Apache Airflow Issue Type: New Feature Components: contrib Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 1.10.2 Currently, there is no provision in Airflow's implementation of SSHOperator to disable Host Checking and use a Null Known Host file to allow connecting to host that has a new IP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2104) Recommendation for Airflow design/architecture
[ https://issues.apache.org/jira/browse/AIRFLOW-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-2104. --- Resolution: Not A Problem > Recommendation for Airflow design/architecture > -- > > Key: AIRFLOW-2104 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2104 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, configuration, DagRun >Affects Versions: 1.9.0 >Reporter: Anil Kumar >Priority: Major > Fix For: 1.9.0 > > > Hello, > We are in the implementation phase for Apache Airflow and looking for > Recommendation on Airflow configuration/structure. > Which one will be best option > # Airflow standalone installation with multi worker nodes or > # Airflow running as docker image on Kubernetes > Requirement :- > We are suppose to schedule a long running ETL process that includes complex > unzip/decrypt task, execution of data wrangling UDF's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3396) Redundant sql param check in BigQueryOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3396: Description: The 'sql' parameter in BigQueryOperator accepts None and then there is a check which checks if it is None and gives error. (was: The 'sql' p) > Redundant sql param check in BigQueryOperator > - > > Key: AIRFLOW-3396 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3396 > Project: Apache Airflow > Issue Type: Task > Components: gcp >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Trivial > Fix For: 2.0.0 > > > The 'sql' parameter in BigQueryOperator accepts None and then there is a > check which checks if it is None and gives error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3396) Redundant sql param check in BigQueryOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3396: Description: The 'sql' p > Redundant sql param check in BigQueryOperator > - > > Key: AIRFLOW-3396 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3396 > Project: Apache Airflow > Issue Type: Task > Components: gcp >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Trivial > Fix For: 2.0.0 > > > The 'sql' p -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3396) Redundant sql param check in BigQueryOperator
Kaxil Naik created AIRFLOW-3396: --- Summary: Redundant sql param check in BigQueryOperator Key: AIRFLOW-3396 URL: https://issues.apache.org/jira/browse/AIRFLOW-3396 Project: Apache Airflow Issue Type: Task Components: gcp Reporter: Kaxil Naik Assignee: Kaxil Naik Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-156) Add date option to trigger_dag
[ https://issues.apache.org/jira/browse/AIRFLOW-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-156. -- Resolution: Duplicate > Add date option to trigger_dag > -- > > Key: AIRFLOW-156 > URL: https://issues.apache.org/jira/browse/AIRFLOW-156 > Project: Apache Airflow > Issue Type: New Feature > Components: cli >Reporter: Jeffrey Picard >Priority: Minor > > Currently the trigger_dag command always sets the execution date to > datetime.now(). This seems like a rather arbitrary restriction and there > are use cases when running dags ad-hoc where one may wish to set a > different execution date. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
