[GitHub] [airflow] turbaszek commented on issue #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor
turbaszek commented on issue #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor URL: https://github.com/apache/airflow/pull/7731#issuecomment-607051121 > See also #7541 (which I'm in the process of fixing up) - it shouldn't be a big change for you to rebase on to that once I have it passing/merged. @ashb should we merge or wait? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #8035: Security: make user in mysql
zhongjiajie commented on issue #8035: Security: make user in mysql URL: https://github.com/apache/airflow/issues/8035#issuecomment-607046659 Try to use `airflow users create` in command line to create new user? @konia-s This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8037: Update instructions to prepare backport packages
turbaszek commented on a change in pull request #8037: Update instructions to prepare backport packages URL: https://github.com/apache/airflow/pull/8037#discussion_r401367286 ## File path: CONTRIBUTING.rst ## @@ -998,10 +998,12 @@ prepare such packages on your own easily. * Enter the ``backport_packages`` directory +* Run ``python setup_backport_packages.py prepare`` to update deprecated modules & paths. + * Usually you only build some of the providers package. The ``providers`` directory is separated into separate providers. You can see the list of all available providers by running ``python setup_backport_packages.py list-backport-packages``. You can build the backport package - by running ``python setup.py bdist_wheel``. Note that there + by running ``python setup_backport_packages.py bdist_wheel``. Note that there Review comment: I think we should encourage people to use one-step-only: ``` scripts/ci/ci_prepare_packages.sh packageA packageB packageC ``` WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on issue #6379: [AIRFLOW-5708] Optionally check task pools when parsing dags.
jmcarp commented on issue #6379: [AIRFLOW-5708] Optionally check task pools when parsing dags. URL: https://github.com/apache/airflow/pull/6379#issuecomment-607020397 Updated, cc @mik-laj @Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7832: Add production image support
codecov-io edited a comment on issue #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#issuecomment-605517449 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=h1) Report > Merging [#7832](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26b05bfe36d131ab19744514f9c9c69e9df126aa?src=pr=desc) will **increase** coverage by `0.09%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7832 +/- ## == + Coverage 87.16% 87.26% +0.09% == Files 932 932 Lines 4533145331 == + Hits3951439559 +45 + Misses 5817 5772 -45 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7832/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.94% <0%> (+0.57%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7832/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.64% <0%> (+1.11%)` | :arrow_up: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7832/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0%> (+4.89%)` | :arrow_up: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/7832/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `100% <0%> (+44%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=footer). Last update [26b05bf...182a141](https://codecov.io/gh/apache/airflow/pull/7832?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] eladkal commented on issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0
eladkal commented on issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0 URL: https://github.com/apache/airflow/issues/8036#issuecomment-606988925 was fixed in master https://github.com/apache/airflow/pull/7857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins edited comment on AIRFLOW-6515 at 4/1/20, 1:26 AM:
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, correct?
was (Author: wkhudgins):
[~toopt4] Sounds like this is essentially going through those and similar files
and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
>
[jira] [Comment Edited] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins edited comment on AIRFLOW-6515 at 4/1/20, 1:26 AM:
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, and if it doesn't, fixing the
log level, yes?
was (Author: wkhudgins):
[~toopt4] Sounds like this is essentially going through those and similar log
calls and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base,
[jira] [Commented] (AIRFLOW-6515) log level of INFO/WARN when ERROR happened
[
https://issues.apache.org/jira/browse/AIRFLOW-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072290#comment-17072290
]
Will Hudgins commented on AIRFLOW-6515:
---
[~toopt4] Sounds like this is essentially going through those and similar files
and making sure the log level makes sense, correct?
> log level of INFO/WARN when ERROR happened
> --
>
> Key: AIRFLOW-6515
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6515
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> log level should be error on some (but there are false positives):
> grep -iE
> 'log\.(info|warn).*(error|exceptio|fail|unab|couldn|lost|gone|missing|not
> fou|abort|exit|could not)' -R *
> airflow/sensors/base_sensor_operator.py:self.log.info("Success
> criteria met. Exiting.")
> airflow/logging_config.py:log.warning('Unable to load the config,
> contains a configuration error.')
> airflow/operators/check_operator.py:self.log.warning("The
> following %s tests out of %s failed:", j, n)
> airflow/operators/sql_to_gcs.py:self.log.warning('Using default
> schema due to missing name or type. Please '
> airflow/operators/bash_operator.py:self.log.info('Command exited
> with return code %s', self.sub_process.returncode)
> airflow/serialization/serialized_objects.py:LOG.warning('Failed
> to stringify.', exc_info=True)
> airflow/providers/amazon/aws/operators/batch.py:
> self.log.info("AWS Batch Job has failed")
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/providers/amazon/aws/hooks/s3.py:
> self.log.info(e.response["Error"]["Message"])
> airflow/utils/dag_processing.py:self.log.info("Exiting gracefully
> upon receiving signal %s", signum)
> airflow/utils/dag_processing.py:self.log.info("Exiting dag
> parsing loop as all files "
> airflow/utils/dag_processing.py:self.log.info("Failing jobs
> without heartbeat after %s", limit_dttm)
> airflow/utils/dag_processing.py:self.log.info("Waiting up to %s
> seconds for processes to exit...", timeout)
> airflow/utils/helpers.py:log.info("Process %s (%s) terminated with
> exit code %s", p, p.pid, p.returncode)
> airflow/models/dagrun.py:self.log.info('Marking run %s failed',
> self)
> airflow/models/dagrun.py:self.log.info('Deadlock; marking run %s
> failed', self)
> airflow/models/dagrun.py:self.log.warning("Failed to get
> task '{}' for dag '{}'. "
> airflow/gcp/sensors/gcs.py:self.log.warning("FAILURE: Inactivity
> Period passed, not enough objects found in %s", path)
> airflow/gcp/operators/spanner.py:self.log.info("The Cloud Spanner
> database was missing: "
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/kubernetes_engine.py:self.log.info('Assuming
> Success: %s', error.message)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Failovering
> Instance: %s", name)
> airflow/gcp/hooks/cloud_memorystore.py:self.log.info("Instance
> failovered: %s", name)
> airflow/gcp/hooks/bigquery.py:self.log.info(error_msg)
> airflow/gcp/hooks/bigtable.py:self.log.info("The instance '%s'
> does not exist in project '%s'. Exiting", instance_id,
> airflow/contrib/sensors/bash_sensor.py:self.log.info("Command
> exited with return code %s", sp.returncode)
> airflow/contrib/sensors/ftp_sensor.py:self.log.info('Ftp
> error encountered: %s', str(e))
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container had exit code: %s", exit_code)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Container exited with detail_status %s", detail_status)
> airflow/contrib/operators/azure_container_instances_operator.py:
> self.log.info("Azure provision failure")
> airflow/contrib/operators/winrm_operator.py:self.log.info("Hook
> not found, creating...")
> airflow/contrib/operators/docker_swarm_operator.py:
> self.log.info('Service status before exiting: %s', status)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Unable to
> find group for %s %s", search_base, search_filter)
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("""Missing
> attribute "%s" when looked-up in Ldap database.
> airflow/contrib/auth/backends/ldap_auth.py:log.warning("Parsing error
> when retrieving the user's group(s)."
>
[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072289#comment-17072289 ] Will Hudgins commented on AIRFLOW-6281: --- [~kamil.bregula] This is for the GCSToGCSOperator in [https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/gcs_to_gcs.py] ? If so I'd be willing to take it on. > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6295) Create guide for BigQuery service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072288#comment-17072288 ] Will Hudgins commented on AIRFLOW-6295: --- Beat me to it [~alexaabbas]!!! > Create guide for BigQuery service operator > -- > > Key: AIRFLOW-6295 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6295 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Assignee: Alexandra Abbas >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use BigQuery service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1135) Clearing XCom data from previous executions
[ https://issues.apache.org/jira/browse/AIRFLOW-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072277#comment-17072277 ] Chris commented on AIRFLOW-1135: Yes, [~ultrabug] stated precisely a common use case. I found this Stackoverflow which could do this manually, but having a flag in the xcom pull to delete the xcom once accessed would be very useful. > Clearing XCom data from previous executions > --- > > Key: AIRFLOW-1135 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1135 > Project: Apache Airflow > Issue Type: New Feature > Components: database >Reporter: Ultrabug >Priority: Major > > It looks like XCom data from previous executions of tasks are never cleared > from database and that there's no easy way to do so. > Since XComs are created by execution_date of a task, when a task heavily uses > XComs (like 20 000+) we can end up with a large xcom table on the database. > Is there something I'm missing about cleaning up those previous XComs from > database (since they're not used anymore) ? > Would you accept the idea of having a new PR providing a helper to remove > them ? > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-353) when multiple tasks removed update state fails
[ https://issues.apache.org/jira/browse/AIRFLOW-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072273#comment-17072273 ] Chris commented on AIRFLOW-353: --- I'm experiencing https://issues.apache.org/jira/browse/AIRFLOW-7116 which seems related to this. > when multiple tasks removed update state fails > -- > > Key: AIRFLOW-353 > URL: https://issues.apache.org/jira/browse/AIRFLOW-353 > Project: Apache Airflow > Issue Type: Bug > Components: core >Reporter: Yiqing Jin >Assignee: Yiqing Jin >Priority: Major > Fix For: 1.8.0 > > > if multiple tasks gets removed during dag run update_state may not work > properly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1460) Tasks get stuck in the "removed" state
[ https://issues.apache.org/jira/browse/AIRFLOW-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072276#comment-17072276 ] Chris commented on AIRFLOW-1460: Still getting this in 1.10.9: Here are my repro steps: https://issues.apache.org/jira/browse/AIRFLOW-7116 > Tasks get stuck in the "removed" state > -- > > Key: AIRFLOW-1460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1460 > Project: Apache Airflow > Issue Type: Bug >Reporter: George Leslie-Waksman >Assignee: Maxime Beauchemin >Priority: Major > Fix For: 1.10.0 > > > The current handling of task instances that get assigned the state "removed" > is that they get ignored. > If the underlying task later gets re-added, the existing task instances will > prevent the task from running in the future. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7116) Tasks get stuck in the "removed" state
Chris created AIRFLOW-7116: -- Summary: Tasks get stuck in the "removed" state Key: AIRFLOW-7116 URL: https://issues.apache.org/jira/browse/AIRFLOW-7116 Project: Apache Airflow Issue Type: Bug Components: DAG Affects Versions: 1.10.9 Environment: Mac OS Catalina Airflow 1.10.9 Airflow running locally using out of box sqlite database Reporter: Chris Attachments: Untitled.png This appears to be a regression of: https://issues.apache.org/jira/browse/AIRFLOW-1460 Repro steps: # Configure Dag to run within context manager via "with" keyword. # Comment out one or more tasks within a Dag. # Refresh Dag. # Trigger Dag. # Notice the tasks are removed from the dag and the dag runs the non-commented task(s) successfully. # Uncomment the tasks that were commented before. # refresh dag. # trigger dag. Observed results: * The tree view of the dag does not update, showing the re-added tasks. * After Triggering the Dag, (only the task(s) that was/were tested while the others were commented out appears in the Graph View) * On the graph view, the dag's uncommented tasks only appear after restarting Airflow Server/Scheduler. * Even though the Airflow server was restarted, and although the dag now shows all the tasks in both Tree view and Graph view, the dag, when triggered, only runs the task(s) that were previously not commented. * Note that the tasks when hovering mouse over them in Graph View, show "State" as "removed". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py
codecov-io edited a comment on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py URL: https://github.com/apache/airflow/pull/8038#issuecomment-606960674 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=h1) Report > Merging [#8038](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8c2bd91a8fb68a36e1190ad019b1f050e4a15c35=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8038 +/- ## == - Coverage 87.27% 86.52% -0.75% == Files 932 934 +2 Lines 4534245357 +15 == - Hits3957039243 -327 - Misses 5772 6114 +342 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: | | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=footer). Last update [8c2bd91...e3aa38b](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py
codecov-io edited a comment on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py URL: https://github.com/apache/airflow/pull/8038#issuecomment-606960674 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=h1) Report > Merging [#8038](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8c2bd91a8fb68a36e1190ad019b1f050e4a15c35=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8038 +/- ## == - Coverage 87.27% 86.52% -0.75% == Files 932 934 +2 Lines 4534245357 +15 == - Hits3957039243 -327 - Misses 5772 6114 +342 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: | | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=footer). Last update [8c2bd91...e3aa38b](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py
codecov-io commented on issue #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py URL: https://github.com/apache/airflow/pull/8038#issuecomment-606960674 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=h1) Report > Merging [#8038](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8c2bd91a8fb68a36e1190ad019b1f050e4a15c35=desc) will **decrease** coverage by `0.74%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8038 +/- ## == - Coverage 87.27% 86.52% -0.75% == Files 932 934 +2 Lines 4534245357 +15 == - Hits3957039243 -327 - Misses 5772 6114 +342 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: | | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/8038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=footer). Last update [8c2bd91...e3aa38b](https://codecov.io/gh/apache/airflow/pull/8038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jvaesteves opened a new issue #8039: KubernetesPodOperator on EKS ignores both user on kubeconfig and service_account_name
jvaesteves opened a new issue #8039: KubernetesPodOperator on EKS ignores both
user on kubeconfig and service_account_name
URL: https://github.com/apache/airflow/issues/8039
**Apache Airflow version**: 1.10.9
**Kubernetes version (if you are using kubernetes)**: client v1.17.2 |
server v1.15.10-eks-bac369
**Environment**: production
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Amazon Linux 2
- **Kernel** (e.g. `uname -a`): Linux 0 4.14.138
- **Install tools**: pip
- **Others**: aws-iam-authenticator
**What happened**:
When I deploy the Airflow pod on my cluster, using attaching AWS credentials
to the pod via secrets, on my entrypoint script, I run `aws eks
update-kubeconfig` to generate a kubeconfig file for it.
The credential only has access to the **airflow** namespace, where every
operation that it performs takes place. I executed a `kubectl run --image
worker-image test` just to be sure that this user has pod creation privillege.
But when I run the KubernetesPodOperator as follows, the client accuses
Forbidden error:
```python
KubernetesPodOperator(
task_id='task_name',
dag=dag,
name='worker-pod-nane',
namespace="airflow",
image="worker-image",
image_pull_policy="Always",
cmds=['python'],
arguments=['task.py'],
)
```
Error:
```javascript
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"system:serviceaccount:airflow:default\" cannot create
resource \"pods\" in API group \"\" in the namespace
\"airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
To mitigate this error, I created an **airflow** ServiceAccount with
full-access to the namespace and added as a parameter `service_account_name` to
the operator but the error was still the same, ignoring the new user name that
I used.
**What you expected to happen**:
I wanted the operator to create a pod on the namespace using its AWS
credentials instead of the defaullt service account. If it is not possible, I
want to know why it is ignoring new serviceaccount that I passed as a parameter
to the operator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8039: KubernetesPodOperator on EKS ignores both user on kubeconfig and service_account_name
boring-cyborg[bot] commented on issue #8039: KubernetesPodOperator on EKS ignores both user on kubeconfig and service_account_name URL: https://github.com/apache/airflow/issues/8039#issuecomment-606953271 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #8033: Unify Google class/package names
mik-laj merged pull request #8033: Unify Google class/package names URL: https://github.com/apache/airflow/pull/8033 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py
kaxil opened a new pull request #8038: Fix typos in scripts/perf/scheduler_dag_execution_timing.py URL: https://github.com/apache/airflow/pull/8038 ShortCircutExecutorMixin -> ShortCircuitExecutorMixin ShortCircutExecutor -> ShortCircuitExecutor --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #8037: Update instructions to prepare backport packages
kaxil opened a new pull request #8037: Update instructions to prepare backport packages URL: https://github.com/apache/airflow/pull/8037 Update instructions to prepare backport packages --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] abhijit-kottur commented on issue #7935: scheduler gets stuck without a trace
abhijit-kottur commented on issue #7935: scheduler gets stuck without a trace URL: https://github.com/apache/airflow/issues/7935#issuecomment-606926354 @mik-laj PostgreSQL. Thats running as a pod too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6802) scheduler not honoring dag.max_active_run config
[ https://issues.apache.org/jira/browse/AIRFLOW-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072177#comment-17072177 ] ASF subversion and git services commented on AIRFLOW-6802: -- Commit 898651f17e7f58246efeb1bd33d98a39b8db3eb5 in airflow's branch refs/heads/v1-10-test from QP Hou [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=898651f ] [AIRFLOW-6802] Fix bug where dag.max_active_run wasn't always honored by scheduler (#7416) commit 50efda5c69c1ddfaa869b408540182fb19f1a286 introduced a bug that prevents the scheduler from enforcing max active run config for all DAGs. this commit fixes the regression as well as the test. (cherry picked from commit 343d38af380afad2b202838317a47a7b1687f14f) > scheduler not honoring dag.max_active_run config > > > Key: AIRFLOW-6802 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6802 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Critical > Fix For: 1.10.10 > > > commit > https://github.com/apache/airflow/commit/50efda5c69c1ddfaa869b408540182fb19f1a286 > introduced a bug that prevents the scheduler from enforcing max active run > config for all DAGs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072176#comment-17072176 ] ASF subversion and git services commented on AIRFLOW-3607: -- Commit 9f93580131280590c28daa921118561bec74f55b in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9f93580 ] [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751) This decreases scheduler delay between tasks by about 20% for larger DAGs, sometimes more for larger or more complex DAGs. The delay between tasks can be a major issue, especially when we have dags with many subdags, figures out that the scheduling process spends plenty of time in dependency checking, we took the trigger rule dependency which calls the db for each task instance, we made it call the db just once for each dag_run > Decreasing scheduler delay between tasks > > > Key: AIRFLOW-3607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3607 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.0, 1.10.1, 1.10.2 > Environment: ubuntu 14.04 >Reporter: Amichai Horvitz >Assignee: Amichai Horvitz >Priority: Major > Fix For: 1.10.10 > > Original Estimate: 336h > Remaining Estimate: 336h > > I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) > that says instead of checking the query for every task let the tasks report > to the dagrun. I have a dag with many tasks and the delay between tasks can > rise to 10 seconds or more, I already changed the configuration, added > processes and memory, checked the code and did research, profiling and other > experiments. I hope that this change will make a drastic change in the delay. > I would be happy to discuss this solution, the research and other solutions > for this issue. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6730) is_alive uses seconds and not total_seconds
[ https://issues.apache.org/jira/browse/AIRFLOW-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072171#comment-17072171 ] ASF subversion and git services commented on AIRFLOW-6730: -- Commit 637b0b9af33a1c301219d02c5534dcd334f75eae in airflow's branch refs/heads/v1-10-test from Alex Guziel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=637b0b9 ] [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) * [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4bab14222da068b737d6332db4963b994007) > is_alive uses seconds and not total_seconds > --- > > Key: AIRFLOW-6730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6730 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.4 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > Example: > timedelta(days=1).seconds = 0 > timedelta(days=1).total_seconds() = 86400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6730) is_alive uses seconds and not total_seconds
[ https://issues.apache.org/jira/browse/AIRFLOW-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072172#comment-17072172 ] ASF subversion and git services commented on AIRFLOW-6730: -- Commit 637b0b9af33a1c301219d02c5534dcd334f75eae in airflow's branch refs/heads/v1-10-test from Alex Guziel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=637b0b9 ] [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) * [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4bab14222da068b737d6332db4963b994007) > is_alive uses seconds and not total_seconds > --- > > Key: AIRFLOW-6730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6730 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.4 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > Example: > timedelta(days=1).seconds = 0 > timedelta(days=1).total_seconds() = 86400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7080) API Endpoint to query a dag's paused status
[ https://issues.apache.org/jira/browse/AIRFLOW-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072167#comment-17072167 ] ASF subversion and git services commented on AIRFLOW-7080: -- Commit 4d20918a9fa7371ecdd1edb47efdd98748c7462c in airflow's branch refs/heads/v1-10-test from Philipp Großelfinger [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4d20918 ] [AIRFLOW-7080] Adds API endpoint to return a DAG's paused state (#7737) Adds an additional endpoint to the experimental API to return the paused state of a DAG. > API Endpoint to query a dag's paused status > --- > > Key: AIRFLOW-7080 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7080 > Project: Apache Airflow > Issue Type: New Feature > Components: api >Affects Versions: 2.0.0, 1.10.10 >Reporter: Philipp Großelfinger >Assignee: Philipp Großelfinger >Priority: Minor > Fix For: 1.10.10 > > > So far it is possible to set the paused state of a DAG via the experimental > API. It would be nice to be also able to query the current paused state of a > DAG. > The endpoint to set the paused state looks like this: > /api/experimental/dags//paused/ > The new endpoint could look like: > GET /api/experimental/dags//paused -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6628) Dag search autocomplete is filtered by page context
[ https://issues.apache.org/jira/browse/AIRFLOW-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072168#comment-17072168 ] ASF subversion and git services commented on AIRFLOW-6628: -- Commit 2e86ea138334decdfa54ea3cb12050620c0ceb73 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2e86ea1 ] [AIRFLOW-6628] DAG auto-complete now suggests from all acessible DAGs (#7251) The auto complete on the dag search box utilises the page search context which prevents auto completing dags which dont match the current query. On the dags page with no search provided it executes a query that loads every dag_id in the system to pass to the typeahead widget. On systems with a lot of dags this makes the page large and slow. Add a JSON endpoint to provide correct auto complete behaviour and reduce page load time. cherry-picked from 1a9d21861 > Dag search autocomplete is filtered by page context > --- > > Key: AIRFLOW-6628 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6628 > Project: Apache Airflow > Issue Type: Bug > Components: ui, webserver >Affects Versions: 2.0.0, 1.10.7, 1.10.10 >Reporter: Robin Edwards >Assignee: Robin Edwards >Priority: Minor > Fix For: 1.10.10 > > > The auto complete on the dag search box utilises the page search context > which prevents it auto completing dags which dont match the current query > Further still on the home page with no search provided it executes a query > that loads every dag_id in the system for it to be passed to the typeahead > widget. On systems with a lot of dags this makes the page slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6167) Escape col name in MysqlToHive operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072170#comment-17072170 ] ASF subversion and git services commented on AIRFLOW-6167: -- Commit e85b0def44c0fd28aa5042b06349a176817ade4e in airflow's branch refs/heads/v1-10-test from Ping Zhang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e85b0de ] [AIRFLOW-6167] Escape column name in create table in hive (#6741) Hive query will fail if column name contains keyword, unless escaped. Fix is always escaping column name when creating table. (cherry picked from commit a1f3b29344d400aefd4a6c1f6842d2a345de) > Escape col name in MysqlToHive operator > --- > > Key: AIRFLOW-6167 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6167 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.4 >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5590) Add 'run_id' to trigger DAG run API response as a field
[
https://issues.apache.org/jira/browse/AIRFLOW-5590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072166#comment-17072166
]
ASF subversion and git services commented on AIRFLOW-5590:
--
Commit 11eb53938359c6f27e5a9c59e9dc8223d62d in airflow's branch
refs/heads/v1-10-test from Kumpan Anton
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=11eb539 ]
[AIRFLOW-5590] Add run_id to trigger DAG run API response (#6256)
> Add 'run_id' to trigger DAG run API response as a field
> ---
>
> Key: AIRFLOW-5590
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5590
> Project: Apache Airflow
> Issue Type: Improvement
> Components: api
>Affects Versions: 1.10.5
>Reporter: Anton Kumpan
>Assignee: Anton Kumpan
>Priority: Major
> Fix For: 1.10.10
>
>
> Idea is the same as in AIRFLOW-4482
> Currently API response looks like this:
> {code:json}
> {
> 'execution_date': '2019-05-08T07:03:09+00:00',
> 'message': 'Created manual__2019-05-08T07:03:09+00:00, externally triggered: True>'
> }
> {code}
>
> It would be nice to add run_id as a separate field, so that response will
> look like:
> {code:json}
> {
> 'run_id': 'manual__2019-05-08T07:03:09+00:00',
> 'execution_date': '2019-05-08T07:03:09+00:00',
> 'message': 'Created triggered: True>'
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6837) limit description of dag in home page
[ https://issues.apache.org/jira/browse/AIRFLOW-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072169#comment-17072169 ] ASF subversion and git services commented on AIRFLOW-6837: -- Commit ae30e4eb04567bf4070845480befaf96985cff21 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=ae30e4e ] [AIRFLOW-6837] Limit description length of a Dag on HomePage (#7457) cherry-picked from 0ce2a55c7 > limit description of dag in home page > - > > Key: AIRFLOW-6837 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6837 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Fix For: 1.10.10 > > > so that the tooltip will not cover the dag link -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7017) Respect default dag view on redirect after trigger
[ https://issues.apache.org/jira/browse/AIRFLOW-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072173#comment-17072173 ] ASF subversion and git services commented on AIRFLOW-7017: -- Commit 43458686f2a2a1e7bbfe2858c2fb38fc6a2efbbc in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4345868 ] [AIRFLOW-7017] Respect default dag view in trigger dag origin (#7667) > Respect default dag view on redirect after trigger > -- > > Key: AIRFLOW-7017 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7017 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Joshua Carp >Priority: Trivial > Fix For: 1.10.10 > > > Triggering a dag from the dag detail page always redirect to the dag tree > view. This redirect should respect the default dag view if configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
randr97 commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401215099
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
+setattr(self, key, val)
+
+return FacebookAdsApi.init(app_id=self.app_id,
Review comment:
Ya this can be done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
randr97 commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401213486
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
Review comment:
Yep, will do so!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb merged pull request #8034: Make DateTimePicker actually pick times too.
ashb merged pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] MatthewRBruce commented on issue #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated
MatthewRBruce commented on issue #7424: [AIRFLOW-6796] Clean up DAG serializations based on last_updated URL: https://github.com/apache/airflow/pull/7424#issuecomment-606870400 It won't delete the DAG until the last time the scheduler processed that DAG has surpassed `dag_cleanup_interval`. So yes, if the the DAG file was deleted, it will take around `dag_cleanup_interval`s for it to be removed. I don't believe the DAG being updated or tasks being added/delete will come into play. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5391) Clearing a task skipped by BranchPythonOperator will cause the task to execute
[ https://issues.apache.org/jira/browse/AIRFLOW-5391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-5391: Fix Version/s: (was: 1.10.10) 2.0.0 > Clearing a task skipped by BranchPythonOperator will cause the task to execute > -- > > Key: AIRFLOW-5391 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5391 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.4 >Reporter: Qian Yu >Assignee: Qian Yu >Priority: Major > Fix For: 2.0.0 > > > I tried this on 1.10.3 and 1.10.4, both have this issue: > E.g. in this example from the doc, branch_a executed, branch_false was > skipped because of branching condition. However if someone Clear > branch_false, it'll cause branch_false to execute. > !https://airflow.apache.org/_images/branch_good.png! > This behaviour is understandable given how BranchPythonOperator is > implemented. BranchPythonOperator does not store its decision anywhere. It > skips its own downstream tasks in the branch at runtime. So there's currently > no way for branch_false to know it should be skipped without rerunning the > branching task. > This is obviously counter-intuitive from the user's perspective. In this > example, users would not expect branch_false to execute when they clear it > because the branching task should have skipped it. > There are a few ways to improve this: > Option 1): Make downstream tasks skipped by BranchPythonOperator not > clearable without also clearing the upstream BranchPythonOperator. In this > example, if someone clears branch_false without clearing branching, the Clear > action should just fail with an error telling the user he needs to clear the > branching task as well. > Option 2): Make BranchPythonOperator store the result of its skip condition > somewhere. Make downstream tasks check for this stored decision and skip > themselves if they should have been skipped by the condition. This probably > means the decision of BranchPythonOperator needs to be stored in the db. > > [kevcampb|https://blog.diffractive.io/author/kevcampb/] attempted a > workaround and on this blog. And he acknowledged his workaround is not > perfect and a better permanent fix is needed: > [https://blog.diffractive.io/2018/08/07/replacement-shortcircuitoperator-for-airflow/] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401208609
##
File path: entrypoint.sh
##
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Might be empty
+AIRFLOW_COMMAND="${1}"
+
+set -euo pipefail
+
+function verify_db_connection {
+DB_URL="${1}"
+
+DB_CHECK_MAX_COUNT=${MAX_DB_CHECK_COUNT:=10}
+DB_CHECK_SLEEP_TIME=${DB_CHECK_SLEEP_TIME:=2}
+
+export DETECTED_DB_BACKEND=""
+export DETECTED_DB_USER=""
+export DETECTED_DB_HOST=""
+export DETECTED_DB_PORT=""
+export DETECTED_DB_SCHEMA=""
+export DETECTED_DB_PARAMS=""
+
+if [[ ${DB_URL} != sqlite* ]]; then
+# Auto-detect DB parameters
+[[ ${DB_URL} =~ ([^:]*)://([^@/]*)@?([^/:]*):?([0-9]*)/([^\?]*)\??(.*)
]] && \
+DETECTED_DB_BACKEND=${BASH_REMATCH[1]} &&
+DETECTED_DB_USER=${BASH_REMATCH[2]} &&
+DETECTED_DB_HOST=${BASH_REMATCH[3]} &&
+DETECTED_DB_PORT=${BASH_REMATCH[4]} &&
+DETECTED_DB_SCHEMA=${BASH_REMATCH[5]} &&
+DETECTED_DB_PARAMS=${BASH_REMATCH[6]}
+
+echo DB_BACKEND="${DB_BACKEND:=${DETECTED_DB_BACKEND}}"
+
+if [[ -z "${DETECTED_DB_PORT}" ]]; then
+if [[ ${DB_BACKEND} == "postgres"* ]]; then
+DETECTED_DB_PORT=5432
+elif [[ ${DB_BACKEND} == "mysql"* ]]; then
+DETECTED_DB_PORT=3306
+fi
+fi
+
+DETECTED_DB_HOST=${DETECTED_DB_HOST:="localhost"}
+
+# Allow the DB parameters to be overridden by environment variable
+echo DB_HOST="${DB_HOST:=${DETECTED_DB_HOST}}"
+echo DB_PORT="${DB_PORT:=${DETECTED_DB_PORT}}"
+
+while true
+do
+set +e
+LAST_CHECK_RESULT=$(nc -zvv "${DB_HOST}" "${DB_PORT}" >/dev/null
2>&1)
+RES=$?
+set -e
+if [[ ${RES} == 0 ]]; then
+echo
+break
+else
+echo -n "."
+DB_CHECK_MAX_COUNT=$((DB_CHECK_MAX_COUNT-1))
+fi
+if [[ ${DB_CHECK_MAX_COUNT} == 0 ]]; then
+echo
+echo "ERROR! Maximum number of retries (${DB_CHECK_MAX_COUNT})
reached while checking ${DB_BACKEND} db. Exiting"
+echo
+break
+else
+sleep "${DB_CHECK_SLEEP_TIME}"
+fi
+done
+if [[ ${RES} != 0 ]]; then
+echo "ERROR: ${BACKEND} db could not be reached!"
+echo
+echo "${LAST_CHECK_RESULT}"
+echo
+export EXIT_CODE=${RES}
+fi
+fi
+}
+
+# if no DB configured - use sqlite db by default
+AIRFLOW__CORE__SQL_ALCHEMY_CONN="${AIRFLOW__CORE__SQL_ALCHEMY_CONN:="sqlite:///${AIRFLOW_HOME}/airflow.db"}"
+
+verify_db_connection "${AIRFLOW__CORE__SQL_ALCHEMY_CONN}"
+
+AIRFLOW__CELERY__BROKER_URL=${AIRFLOW__CELERY__BROKER_URL:=}
+
+if [[ -n ${AIRFLOW__CELERY__BROKER_URL} ]] && \
+[[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|flower)$ ]]; then
+verify_db_connection "${AIRFLOW__CELERY__BROKER_URL}"
+fi
+
+if [[ ${AIRFLOW__CORE__SQL_ALCHEMY_CONN} == sqlite* ]] && \
+[[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|webserver)$ ]] || \
+[[ ${DB_RESET:="false"} == "true" ]]; then
+if [[ ${AIRFLOW_VERSION} == *1.10* ]]; then
Review comment:
This could match 2.1.10 in future
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401208246
##
File path: entrypoint.sh
##
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Might be empty
+AIRFLOW_COMMAND="${1}"
+
+set -euo pipefail
+
+function verify_db_connection {
+DB_URL="${1}"
+
+DB_CHECK_MAX_COUNT=${MAX_DB_CHECK_COUNT:=10}
+DB_CHECK_SLEEP_TIME=${DB_CHECK_SLEEP_TIME:=2}
+
+export DETECTED_DB_BACKEND=""
+export DETECTED_DB_USER=""
+export DETECTED_DB_HOST=""
+export DETECTED_DB_PORT=""
+export DETECTED_DB_SCHEMA=""
+export DETECTED_DB_PARAMS=""
+
+if [[ ${DB_URL} != sqlite* ]]; then
+# Auto-detect DB parameters
+[[ ${DB_URL} =~ ([^:]*)://([^@/]*)@?([^/:]*):?([0-9]*)/([^\?]*)\??(.*)
]] && \
+DETECTED_DB_BACKEND=${BASH_REMATCH[1]} &&
+DETECTED_DB_USER=${BASH_REMATCH[2]} &&
+DETECTED_DB_HOST=${BASH_REMATCH[3]} &&
+DETECTED_DB_PORT=${BASH_REMATCH[4]} &&
+DETECTED_DB_SCHEMA=${BASH_REMATCH[5]} &&
+DETECTED_DB_PARAMS=${BASH_REMATCH[6]}
+
+echo DB_BACKEND="${DB_BACKEND:=${DETECTED_DB_BACKEND}}"
+
+if [[ -z "${DETECTED_DB_PORT}" ]]; then
+if [[ ${DB_BACKEND} == "postgres"* ]]; then
+DETECTED_DB_PORT=5432
+elif [[ ${DB_BACKEND} == "mysql"* ]]; then
+DETECTED_DB_PORT=3306
+fi
+fi
+
+DETECTED_DB_HOST=${DETECTED_DB_HOST:="localhost"}
+
+# Allow the DB parameters to be overridden by environment variable
+echo DB_HOST="${DB_HOST:=${DETECTED_DB_HOST}}"
+echo DB_PORT="${DB_PORT:=${DETECTED_DB_PORT}}"
+
+while true
+do
+set +e
+LAST_CHECK_RESULT=$(nc -zvv "${DB_HOST}" "${DB_PORT}" >/dev/null
2>&1)
+RES=$?
+set -e
+if [[ ${RES} == 0 ]]; then
+echo
+break
+else
+echo -n "."
+DB_CHECK_MAX_COUNT=$((DB_CHECK_MAX_COUNT-1))
+fi
+if [[ ${DB_CHECK_MAX_COUNT} == 0 ]]; then
+echo
+echo "ERROR! Maximum number of retries (${DB_CHECK_MAX_COUNT})
reached while checking ${DB_BACKEND} db. Exiting"
+echo
+break
+else
+sleep "${DB_CHECK_SLEEP_TIME}"
+fi
+done
+if [[ ${RES} != 0 ]]; then
+echo "ERROR: ${BACKEND} db could not be reached!"
+echo
+echo "${LAST_CHECK_RESULT}"
+echo
+export EXIT_CODE=${RES}
+fi
+fi
+}
+
+# if no DB configured - use sqlite db by default
+AIRFLOW__CORE__SQL_ALCHEMY_CONN="${AIRFLOW__CORE__SQL_ALCHEMY_CONN:="sqlite:///${AIRFLOW_HOME}/airflow.db"}"
+
+verify_db_connection "${AIRFLOW__CORE__SQL_ALCHEMY_CONN}"
+
+AIRFLOW__CELERY__BROKER_URL=${AIRFLOW__CELERY__BROKER_URL:=}
+
+if [[ -n ${AIRFLOW__CELERY__BROKER_URL} ]] && \
+[[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|flower)$ ]]; then
+verify_db_connection "${AIRFLOW__CELERY__BROKER_URL}"
+fi
+
+if [[ ${AIRFLOW__CORE__SQL_ALCHEMY_CONN} == sqlite* ]] && \
+[[ ${AIRFLOW_COMMAND} =~ ^(scheduler|worker|webserver)$ ]] || \
+[[ ${DB_RESET:="false"} == "true" ]]; then
+if [[ ${AIRFLOW_VERSION} == *1.10* ]]; then
+airflow reset_db -y >/dev/null 2>&1
Review comment:
Woah. This shouldn't even be an option in a production image, it seems
wildly dangerous.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401206953
##
File path: entrypoint.sh
##
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Might be empty
+AIRFLOW_COMMAND="${1}"
+
+set -euo pipefail
+
+function verify_db_connection {
+DB_URL="${1}"
+
+DB_CHECK_MAX_COUNT=${MAX_DB_CHECK_COUNT:=10}
+DB_CHECK_SLEEP_TIME=${DB_CHECK_SLEEP_TIME:=2}
+
+export DETECTED_DB_BACKEND=""
+export DETECTED_DB_USER=""
+export DETECTED_DB_HOST=""
+export DETECTED_DB_PORT=""
+export DETECTED_DB_SCHEMA=""
+export DETECTED_DB_PARAMS=""
Review comment:
```suggestion
DETECTED_DB_BACKEND=""
DETECTED_DB_USER=""
DETECTED_DB_HOST=""
DETECTED_DB_PORT=""
DETECTED_DB_SCHEMA=""
DETECTED_DB_PARAMS=""
```
These aren't used by any subcommands, so no need to export them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-4425) Add FacebookAdsHook
[ https://issues.apache.org/jira/browse/AIRFLOW-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omair Khan reassigned AIRFLOW-4425: --- Assignee: Omair Khan > Add FacebookAdsHook > --- > > Key: AIRFLOW-4425 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4425 > Project: Apache Airflow > Issue Type: Wish > Components: hooks >Reporter: jack >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Add hook to interact with FacebookAds -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401205424
##
File path: Dockerfile
##
@@ -0,0 +1,373 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALPHA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#much smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
+ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+ARG
CONSTRAINT_REQUIREMENTS="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV CONSTRAINT_REQUIREMENTS=${CONSTRAINT_REQUIREMENTS}
+
+ARG
ENTRYPOINT_FILE="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/entrypoint.sh"
+ENV ENTRYPOINT_FILE="${ENTRYPOINT_FILE}"
+
+# Print versions
+RUN echo "Building airflow-build-image stage" \
+echo "Base image: ${PYTHON_BASE_IMAGE}"; \
+echo "Airflow version: ${AIRFLOW_VERSION}"; \
+echo "Airflow git reference: ${AIRFLOW_GIT_REFERENCE}"; \
+echo
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#discussion_r401204155 ## File path: IMAGES.rst ## @@ -0,0 +1,389 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +.. contents:: :local: + +Airflow docker images += + +Airflow has two images (build from Dockerfiles): + +* CI image (Dockerfile.ci) - used for running tests and local development +* Production image (Dockerfile) - used to run production-ready Airflow installations + +Image naming conventions + + +The images are named as follows: + +``apache/airflow:-python[-ci][-manifest]`` + +where: + +* BRANCH_OR_TAG - branch or tag used when creating the image. Examples: master, v1-10-test, 1.10.10 + The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. the 1.10.* and in + the future ``2.*`` labels are build from git tags and they are "fixed" once built. +* PYTHON_MAJOR_MINOR_VERSION - version of python used to build the image. Examples: 3.5, 3.7 +* The ``-ci`` suffix is added for CI images +* The ``-manifest`` is added for manifest images (see below for explanation of manifest images) + +Building docker images +== + +The easiest way to build those images is to use ``_. + +You can build the CI image using this command: + +.. code-block:: + + ./breeze build-image + +You can build production image using this command: + +.. code-block:: + + ./breeze build-image --production-image + +By adding ``--python `` parameter you can build the +image version for the chosen python version. + +The images are build with default extras - different extras for CI and production image and you +can change the extras via the ``--extras`` parameters. You can see default extras used via +``./breeze flags``. + +For example if you want to build python 3.7 version of production image with +"all" extras installed you should run this command: + +.. code-block:: + + ./breeze build-image --python 3.7 --extras "all" --production-image + +The command that builds the CI image is optimized to minimize the time needed to rebuild the image when +the source code of Airflow evolves. This means that if you already have the image locally downloaded and +built, the scripts will determine whether the rebuild is needed in the first place. Then the scripts will +make sure that minimal number of steps are executed to rebuild parts of the image (for example, +PIP dependencies) and will give you an image consistent with the one used during Continuous Integration. + +The command that build the production image are optimised for size of the image. + +In Breeze by default, the images are built using local sources of Apache Airflow. However +you can also build production images from github sources - providing ``--install-airflow-version`` +parameter to Breeze. This will install airflow inside the production image using sources downloaded from +specified tag or branch. Internally airflow will be installed using the command: + +.. code-block:: + +pip install https://github.com/apache/airflow/archive/>.tar.gz#egg=apache-airflow \ + --constraint https://raw.githubusercontent.com/apache/airflow//requirements/requirements-python3.7.txt + + +Technical details of Airflow images +=== + +The CI image is used by Breeze as shell image but it is also used during CI builds on Travis. +The image is single segment image that contains Airflow installation with "all" dependencies installed. +It is optimised for rebuild speed (AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD flag set to "true"). +It installs PIP dependencies from the current branch first - so that any changes in setup.py do not trigger +reinstalling of all dependencies. There is a second step of installation that re-installs the dependencies +from the latest sources so that we are sure that latest dependencies are installed. + +The production image is a multi-segment image. The first segment "airflow-build-image" contains all the +build essentials and related dependencies that allow to install airflow locally. By default the image
[jira] [Updated] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-1467: Fix Version/s: (was: 2.0.0) 1.10.10 > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 1.10.10 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#discussion_r401203890 ## File path: IMAGES.rst ## @@ -0,0 +1,389 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +.. contents:: :local: + +Airflow docker images += + +Airflow has two images (build from Dockerfiles): + +* CI image (Dockerfile.ci) - used for running tests and local development +* Production image (Dockerfile) - used to run production-ready Airflow installations + +Image naming conventions + + +The images are named as follows: + +``apache/airflow:-python[-ci][-manifest]`` + +where: + +* BRANCH_OR_TAG - branch or tag used when creating the image. Examples: master, v1-10-test, 1.10.10 + The ``master`` and ``v1-10-test`` labels are built from branches so they change over time. the 1.10.* and in + the future ``2.*`` labels are build from git tags and they are "fixed" once built. +* PYTHON_MAJOR_MINOR_VERSION - version of python used to build the image. Examples: 3.5, 3.7 +* The ``-ci`` suffix is added for CI images +* The ``-manifest`` is added for manifest images (see below for explanation of manifest images) + +Building docker images +== + +The easiest way to build those images is to use ``_. + +You can build the CI image using this command: + +.. code-block:: + + ./breeze build-image + +You can build production image using this command: + +.. code-block:: + + ./breeze build-image --production-image + +By adding ``--python `` parameter you can build the +image version for the chosen python version. + +The images are build with default extras - different extras for CI and production image and you +can change the extras via the ``--extras`` parameters. You can see default extras used via +``./breeze flags``. + +For example if you want to build python 3.7 version of production image with +"all" extras installed you should run this command: + +.. code-block:: + + ./breeze build-image --python 3.7 --extras "all" --production-image + +The command that builds the CI image is optimized to minimize the time needed to rebuild the image when +the source code of Airflow evolves. This means that if you already have the image locally downloaded and +built, the scripts will determine whether the rebuild is needed in the first place. Then the scripts will +make sure that minimal number of steps are executed to rebuild parts of the image (for example, +PIP dependencies) and will give you an image consistent with the one used during Continuous Integration. + +The command that build the production image are optimised for size of the image. + +In Breeze by default, the images are built using local sources of Apache Airflow. However +you can also build production images from github sources - providing ``--install-airflow-version`` +parameter to Breeze. This will install airflow inside the production image using sources downloaded from +specified tag or branch. Internally airflow will be installed using the command: + +.. code-block:: + +pip install https://github.com/apache/airflow/archive/>.tar.gz#egg=apache-airflow \ + --constraint https://raw.githubusercontent.com/apache/airflow//requirements/requirements-python3.7.txt + + +Technical details of Airflow images +=== + +The CI image is used by Breeze as shell image but it is also used during CI builds on Travis. +The image is single segment image that contains Airflow installation with "all" dependencies installed. +It is optimised for rebuild speed (AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD flag set to "true"). +It installs PIP dependencies from the current branch first - so that any changes in setup.py do not trigger +reinstalling of all dependencies. There is a second step of installation that re-installs the dependencies +from the latest sources so that we are sure that latest dependencies are installed. + +The production image is a multi-segment image. The first segment "airflow-build-image" contains all the +build essentials and related dependencies that allow to install airflow locally. By default the image
[GitHub] [airflow] kaxil merged pull request #8014: Prevent sequential scan of task instance table when clearing dags
kaxil merged pull request #8014: Prevent sequential scan of task instance table when clearing dags URL: https://github.com/apache/airflow/pull/8014 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=h1) Report > Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26b05bfe36d131ab19744514f9c9c69e9df126aa=desc) will **decrease** coverage by `0.67%`. > The diff coverage is `85.45%`. [](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8008 +/- ## == - Coverage 87.16% 86.49% -0.68% == Files 932 935 +3 Lines 4533145441 +110 == - Hits3951439302 -212 - Misses 5817 6139 +322 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `78.33% <78.33%> (ø)` | | | [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `91.42% <91.42%> (ø)` | | | [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: | | ... and [18 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=footer). Last update [26b05bf...ed5efc0](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-1467) allow tasks to use more than one pool slot
[ https://issues.apache.org/jira/browse/AIRFLOW-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-1467: Fix Version/s: (was: 1.10.10) 2.0.0 > allow tasks to use more than one pool slot > -- > > Key: AIRFLOW-1467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1467 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Reporter: Adrian Bridgett >Assignee: Lokesh Lal >Priority: Trivial > Labels: pool > Fix For: 2.0.0 > > > It would be useful to have tasks use more than a single pool slot. > Our use case is actually to limit how many tasks run on a head node (due to > memory constraints), currently we have to set a pool limit limiting how many > tasks. > Ideally we could set the pool size to e.g amount of memory and then set those > tasks pool_usage (or whatever the option would be called) to the amount of > memory we think they'll use. This way the pool would let lots of small tasks > run or just a few large tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401201166
##
File path: Dockerfile
##
@@ -0,0 +1,373 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALPHA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#much smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
Review comment:
This is quite an important point, and should be discussed on the mailing
list.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator URL: https://github.com/apache/airflow/pull/8008#discussion_r401201092 ## File path: airflow/providers/facebook/ads/hooks/ads.py ## @@ -0,0 +1,150 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains Facebook Ads Reporting hooks +""" +import time +from typing import Any, Dict, List, Optional + +from facebook_business.adobjects.adaccount import AdAccount +from facebook_business.adobjects.adreportrun import AdReportRun +from facebook_business.adobjects.adsinsights import AdsInsights +from facebook_business.api import Cursor, FacebookAdsApi +from facebook_business.exceptions import FacebookRequestError + +from airflow.exceptions import AirflowException +from airflow.hooks.base_hook import BaseHook + +JOB_COMPLETED = 'Job Completed' + + +class FacebookAdsReportingHook(BaseHook): +""" +Hook for the Facebook Ads API + +.. seealso:: +For more information on the Facebook Ads API, take a look at the API docs: +https://developers.facebook.com/docs/marketing-apis/ + +:param facebook_ads_conn_id: Airflow Facebook Ads connection ID +:type facebook_ads_conn_id: str +:param api_version: The version of Facebook API. Default to v6.0 +:type api_version: str + +:return: list of Facebook Ads AdsInsights object(s) +:rtype: list[AdsInsights] Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8014: Prevent sequential scan of task instance table when clearing dags
codecov-io commented on issue #8014: Prevent sequential scan of task instance table when clearing dags URL: https://github.com/apache/airflow/pull/8014#issuecomment-606862646 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=h1) Report > Merging [#8014](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/779023968f983c91701f687bc823dc338934cdad=desc) will **decrease** coverage by `27.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#8014 +/- ## === - Coverage 87.19% 59.86% -27.33% === Files 933 932-1 Lines 4528945331 +42 === - Hits3949027139-12351 - Misses 579918192+12393 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `91.68% <ø> (ø)` | | | [airflow/providers/amazon/aws/hooks/kinesis.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9raW5lc2lzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/providers/apache/livy/sensors/livy.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvc2Vuc29ycy9saXZ5LnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/providers/amazon/aws/sensors/redshift.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL3JlZHNoaWZ0LnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/providers/microsoft/azure/operators/adx.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL29wZXJhdG9ycy9hZHgucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...irflow/providers/amazon/aws/hooks/batch\_waiters.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF93YWl0ZXJzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...ow/providers/amazon/aws/sensors/cloud\_formation.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL2Nsb3VkX2Zvcm1hdGlvbi5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [...w/providers/snowflake/operators/s3\_to\_snowflake.py](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL29wZXJhdG9ycy9zM190b19zbm93Zmxha2UucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [321 more](https://codecov.io/gh/apache/airflow/pull/8014/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=footer). Last update [7790239...e5b8330](https://codecov.io/gh/apache/airflow/pull/8014?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator URL: https://github.com/apache/airflow/pull/8008#discussion_r401201161 ## File path: airflow/providers/facebook/ads/hooks/ads.py ## @@ -0,0 +1,150 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +This module contains Facebook Ads Reporting hooks +""" +import time +from typing import Any, Dict, List, Optional + +from facebook_business.adobjects.adaccount import AdAccount +from facebook_business.adobjects.adreportrun import AdReportRun +from facebook_business.adobjects.adsinsights import AdsInsights +from facebook_business.api import Cursor, FacebookAdsApi +from facebook_business.exceptions import FacebookRequestError + +from airflow.exceptions import AirflowException +from airflow.hooks.base_hook import BaseHook + +JOB_COMPLETED = 'Job Completed' + + +class FacebookAdsReportingHook(BaseHook): +""" +Hook for the Facebook Ads API + +.. seealso:: +For more information on the Facebook Ads API, take a look at the API docs: +https://developers.facebook.com/docs/marketing-apis/ + +:param facebook_ads_conn_id: Airflow Facebook Ads connection ID +:type facebook_ads_conn_id: str +:param api_version: The version of Facebook API. Default to v6.0 +:type api_version: str + +:return: list of Facebook Ads AdsInsights object(s) Review comment: ```suggestion ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401200600
##
File path: Dockerfile
##
@@ -0,0 +1,375 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALFA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#nuch smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
+ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+ARG
CONSTRAINT_REQUIREMENTS="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV CONSTRAINT_REQUIREMENTS=${CONSTRAINT_REQUIREMENTS}
+
+ARG
ENTRYPOINT_FILE="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/entrypoint.sh"
+ENV ENTRYPOINT_FILE="${ENTRYPOINT_FILE}"
+
+# Print versions
+RUN echo "Building airflow-build-image stage" \
+echo "Base image: ${PYTHON_BASE_IMAGE}"; \
+echo "Airflow version: ${AIRFLOW_VERSION}"; \
+echo "Airflow git reference: ${AIRFLOW_GIT_REFERENCE}"; \
+echo
[GitHub] [airflow] ashb commented on issue #7832: Add production image support
ashb commented on issue #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#issuecomment-606860520 I don't see why we want to build from remote sources at all. I can't see where it would be used. Where does the Dockerfile come from to do such a build? If we got it out of the branch/tag for the repo: why not just build form the checkout rather than having to go fetch it again? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401198075
##
File path: airflow/providers/facebook/ads/operators/ads.py
##
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+"""
+Fetches the results from the Facebook Ads API as desired in the params
+Converts and saves the data as a temporary JSON file
+Uploads the JSON to Google Cloud Storage
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+.. seealso::
+For more information on the Facebook Ads Python SDK, take a look at
the docs:
+https://github.com/facebook/facebook-python-business-sdk
+
+:param bucket: The GCS bucket to upload to
+:type bucket: str
+:param obj: GCS path to save the object. Must be the full file path (ex.
`path/to/file.txt`)
+:type obj: str
+:param gcp_conn_id: Airflow Google Cloud Platform connection ID
+:type gcp_conn_id: str
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+:param fields: List of fields that is obtained from Facebook. Found in
AdsInsights.Field class.
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+:type fields: List[str]
+:param params: Parameters that determine the query for Facebook
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+:type params: Dict[str, Any]
+:param gzip: Option to compress local file or file data for upload
+:type gzip: bool
+"""
+
+template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+@apply_defaults
+def __init__(
+self,
+bucket: str,
+obj: str,
Review comment:
```suggestion
object_naame: str,
```
obj is the word reserved.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401198133
##
File path: airflow/providers/facebook/ads/operators/ads.py
##
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+"""
+Fetches the results from the Facebook Ads API as desired in the params
+Converts and saves the data as a temporary JSON file
+Uploads the JSON to Google Cloud Storage
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+.. seealso::
+For more information on the Facebook Ads Python SDK, take a look at
the docs:
+https://github.com/facebook/facebook-python-business-sdk
+
+:param bucket: The GCS bucket to upload to
+:type bucket: str
+:param obj: GCS path to save the object. Must be the full file path (ex.
`path/to/file.txt`)
+:type obj: str
+:param gcp_conn_id: Airflow Google Cloud Platform connection ID
+:type gcp_conn_id: str
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+:param fields: List of fields that is obtained from Facebook. Found in
AdsInsights.Field class.
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+:type fields: List[str]
+:param params: Parameters that determine the query for Facebook
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+:type params: Dict[str, Any]
+:param gzip: Option to compress local file or file data for upload
+:type gzip: bool
+"""
+
+template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+@apply_defaults
+def __init__(
+self,
+bucket: str,
Review comment:
```suggestion
bucket_name: str,
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401198373
##
File path: Dockerfile
##
@@ -0,0 +1,373 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALPHA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#much smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
Review comment:
Just noticed this, should have twigged earlier.
I'm not sure how I feel about installing from git, rather than from the
release artifacts we vote upon.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401197601
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
+setattr(self, key, val)
+
+return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+def _get_config(self) -> None:
+"""
+Gets Facebook ads connection from meta db and sets
+facebook_ads_config attribute with returned config file
+"""
+conn = self.get_connection(self.facebook_ads_conn_id)
+if "facebook_ads_client" not in conn.extra_dejson:
+raise AirflowException("facebook_ads_client not found in extra
field")
+
+self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
+
+def bulk_facebook_report(
+self,
+params: Optional[Dict[str, Any]] = None,
+fields: Optional[List[str]] = None,
+) -> List[AdsInsights]:
+"""
+Pulls data from the Facebook Ads API
+
+:param fields: List of fields that is obtained from Facebook. Found in
AdsInsights.Field class.
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+:type fields: List[str]
+:param params: Parameters that determine the query for Facebook
+
https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+:return: Facebook Ads API response, converted to Facebook Ads Row
objects
+:rtype: List[AdsInsights]
+"""
+api = self._get_service()
+params = params if params else self.default_params
+ad_account = AdAccount(self.ad_account_id, api=api)
+_async = ad_account.get_insights(params=params, fields=fields,
is_async=True)
+try:
+while _async.api_get()["async_status"] != JOB_COMPLETED:
+percent = _async.api_get()["async_percent_completion"]
+self.log.info("%s %s completed", percent, "%")
+
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401196773
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
+setattr(self, key, val)
+
+return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+def _get_config(self) -> None:
+"""
+Gets Facebook ads connection from meta db and sets
+facebook_ads_config attribute with returned config file
+"""
+conn = self.get_connection(self.facebook_ads_conn_id)
+if "facebook_ads_client" not in conn.extra_dejson:
+raise AirflowException("facebook_ads_client not found in extra
field")
+
+self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
Review comment:
What do you think to use cached_property here? It will cause that the
person who wants to use it will not have to know in which order the methods
should be performed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401195975
##
File path: Dockerfile
##
@@ -0,0 +1,375 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALFA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#nuch smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
+ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+ARG
CONSTRAINT_REQUIREMENTS="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV CONSTRAINT_REQUIREMENTS=${CONSTRAINT_REQUIREMENTS}
+
+ARG
ENTRYPOINT_FILE="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/entrypoint.sh"
Review comment:
Official docker images are so you can do `docker pull airflow`? I'm not sure
AFS would like us to drop the Apache name there.
This is an automated message from the Apache Git Service.
To respond
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401195872
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
+setattr(self, key, val)
+
+return FacebookAdsApi.init(app_id=self.app_id,
Review comment:
If these values are from the previous line, is it necessary to not add
additional attributes to the class?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7832: Add production image support
ashb commented on a change in pull request #7832: Add production image support
URL: https://github.com/apache/airflow/pull/7832#discussion_r401195449
##
File path: Dockerfile
##
@@ -0,0 +1,375 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# THIS DOCKERFILE IS INTENDED FOR PRODUCTION USE AND DEPLOYMENT.
+# NOTE! IT IS ALFA-QUALITY FOR NOW - WE ARE IN A PROCESS OF TESTING IT
+#
+#
+# This is a multi-segmented image. It actually contains two images:
+#
+# airflow-build-image - there all airflow dependencies can be installed (and
+#built - for those dependencies that require
+#build essentials). Airflow is installed there with
+#--user switch so that all the dependencies are
+#installed to ${HOME}/.local
+#
+# main - this is the actual production image that is much
+#smaller because it does not contain all the build
+#essentials. Instead the ${HOME}/.local folder
+#is copied from the build-image - this way we have
+#only result of installation and we do not need
+#all the build essentials. This makes the image
+#nuch smaller.
+#
+ARG AIRFLOW_VERSION="2.0.0.dev0"
+ARG AIRFLOW_ORG="apache"
+ARG AIRFLOW_REPO="airflow"
+ARG AIRFLOW_GIT_REFERENCE="master"
+ARG REQUIREMENTS_GIT_REFERENCE="master"
+ARG WWW_FOLDER="www"
+ARG
AIRFLOW_EXTRAS="async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv"
+
+ARG AIRFLOW_HOME=/opt/airflow
+ARG AIRFLOW_UID="5"
+ARG AIRFLOW_GID="5"
+
+ARG PIP_VERSION="19.0.2"
+ARG CASS_DRIVER_BUILD_CONCURRENCY="8"
+
+ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
+ARG PYTHON_MAJOR_MINOR_VERSION="3.6"
+
+##
+# This is the build image where we build all dependencies
+##
+FROM ${PYTHON_BASE_IMAGE} as airflow-build-image
+SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
+
+LABEL org.apache.airflow.docker=true
+LABEL org.apache.airflow.distro="debian"
+LABEL org.apache.airflow.distro.version="buster"
+LABEL org.apache.airflow.module="airflow"
+LABEL org.apache.airflow.component="airflow"
+LABEL org.apache.airflow.image="airflow-build-image"
+LABEL org.apache.airflow.uid="${AIRFLOW_UID}"
+
+ARG AIRFLOW_VERSION
+ARG AIRFLOW_ORG
+ARG AIRFLOW_REPO
+ARG AIRFLOW_GIT_REFERENCE
+ARG REQUIREMENTS_GIT_REFERENCE
+ARG WWW_FOLDER
+ARG AIRFLOW_EXTRAS
+
+ARG AIRFLOW_HOME
+ARG AIRFLOW_UID
+ARG AIRFLOW_GID
+
+ARG PIP_VERSION
+ARG CASS_DRIVER_BUILD_CONCURRENCY
+
+ARG PYTHON_MAJOR_MINOR_VERSION
+
+ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE}
+ENV PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+
+ENV AIRFLOW_VERSION=${AIRFLOW_VERSION}
+ENV AIRFLOW_ORG=${AIRFLOW_ORG}
+ENV AIRFLOW_REPO=${AIRFLOW_REPO}
+ENV AIRFLOW_GIT_REFERENCE=${AIRFLOW_GIT_REFERENCE}
+ENV REQUIREMENTS_GIT_REFERENCE=${REQUIREMENTS_GIT_REFERENCE}
+
+ENV AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}
+
+ENV AIRFLOW_REPO_URL="https://github.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+ENV
AIRFLOW_RAW_CONTENT_URL="https://raw.githubusercontent.com/${AIRFLOW_ORG}/${AIRFLOW_REPO};
+
+ENV PIP_VERSION=${PIP_VERSION}
+ENV CASS_DRIVER_BUILD_CONCURRENCY=${CASS_DRIVER_BUILD_CONCURRENCY}
+
+ARG
AIRFLOW_SOURCES="${AIRFLOW_REPO_URL}/archive/${AIRFLOW_GIT_REFERENCE}.tar.gz#egg=apache-airflow"
+ENV AIRFLOW_SOURCES=${AIRFLOW_SOURCES}
+
+ARG
CONSTRAINT_REQUIREMENTS="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV CONSTRAINT_REQUIREMENTS=${CONSTRAINT_REQUIREMENTS}
+
+ARG
ENTRYPOINT_FILE="${AIRFLOW_RAW_CONTENT_URL}/${REQUIREMENTS_GIT_REFERENCE}/entrypoint.sh"
Review comment:
> This way the Dockerfile becomes a "standalone" Dockerfile.
>
> The idea is that once we release Airflow 1.10.10 you should be able to
just take the Dockerfile (and nothing else) put it in an empty folder and run:
This is different to every other
[GitHub] [airflow] mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
mik-laj commented on a change in pull request #8008: Custom Facebook Ads
Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401195142
##
File path: airflow/providers/facebook/ads/hooks/ads.py
##
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+"""
+Hook for the Facebook Ads API
+
+.. seealso::
+For more information on the Facebook Ads API, take a look at the API
docs:
+https://developers.facebook.com/docs/marketing-apis/
+
+:param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+:type facebook_ads_conn_id: str
+:param api_version: The version of Facebook API. Default to v6.0
+:type api_version: str
+
+:return: list of Facebook Ads AdsInsights object(s)
+:rtype: list[AdsInsights]
+"""
+
+def __init__(
+self,
+facebook_ads_conn_id: str = "facebook_ads_default",
+api_version: str = "v6.0",
+) -> None:
+super().__init__()
+self.facebook_ads_conn_id = facebook_ads_conn_id
+self.facebook_ads_config: Dict[str, Any] = {}
+self.api_version = api_version
+self.ad_account_id: str = ""
+self.app_id: str = ""
+self.app_secret: str = ""
+self.access_token: str = ""
+self.default_params = {
+'level': 'ad',
+'date_preset': 'yesterday',
+}
+
+def _get_service(self) -> FacebookAdsApi:
+""" Returns Facebook Ads Client using a service account"""
+self._get_config()
+for key, val in self.facebook_ads_config.items():
Review comment:
It doesn't seem safe. Can you explicitly set attributes here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7832: Add production image support
ashb commented on issue #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#issuecomment-606856606 Looking This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8008: Custom Facebook Ads Operator
codecov-io commented on issue #8008: Custom Facebook Ads Operator URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=h1) Report > Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26b05bfe36d131ab19744514f9c9c69e9df126aa=desc) will **decrease** coverage by `0.90%`. > The diff coverage is `85.45%`. [](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8008 +/- ## == - Coverage 87.16% 86.26% -0.91% == Files 932 935 +3 Lines 4533145441 +110 == - Hits3951439199 -315 - Misses 5817 6242 +425 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `78.33% <78.33%> (ø)` | | | [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `91.42% <91.42%> (ø)` | | | [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: | | ... and [23 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=footer). Last update [26b05bf...ed5efc0](https://codecov.io/gh/apache/airflow/pull/8008?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6872) Git Version is always Not Available in UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6872: Fix Version/s: (was: 1.10.10) 2.0.0 > Git Version is always Not Available in UI > - > > Key: AIRFLOW-6872 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6872 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.7, 1.10.8, 1.10.9 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > > git_version is always "Not Available" in Webserver. This is because the file > (git_version) is missing from the final built package -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6730) is_alive uses seconds and not total_seconds
[ https://issues.apache.org/jira/browse/AIRFLOW-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072116#comment-17072116 ] ASF subversion and git services commented on AIRFLOW-6730: -- Commit 2315fb92424687bf262e0d08a297945562886a65 in airflow's branch refs/heads/v1-10-test from Alex Guziel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2315fb9 ] [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) * [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4bab14222da068b737d6332db4963b994007) > is_alive uses seconds and not total_seconds > --- > > Key: AIRFLOW-6730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6730 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.4 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > Example: > timedelta(days=1).seconds = 0 > timedelta(days=1).total_seconds() = 86400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6730) is_alive uses seconds and not total_seconds
[ https://issues.apache.org/jira/browse/AIRFLOW-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072115#comment-17072115 ] ASF subversion and git services commented on AIRFLOW-6730: -- Commit 2315fb92424687bf262e0d08a297945562886a65 in airflow's branch refs/heads/v1-10-test from Alex Guziel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2315fb9 ] [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) * [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4bab14222da068b737d6332db4963b994007) > is_alive uses seconds and not total_seconds > --- > > Key: AIRFLOW-6730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6730 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.4 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > Example: > timedelta(days=1).seconds = 0 > timedelta(days=1).total_seconds() = 86400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7017) Respect default dag view on redirect after trigger
[ https://issues.apache.org/jira/browse/AIRFLOW-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072117#comment-17072117 ] ASF subversion and git services commented on AIRFLOW-7017: -- Commit 21ed842b9616c2c56d5d842abf4bf62f228431f4 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=21ed842 ] [AIRFLOW-7017] Respect default dag view in trigger dag origin (#7667) > Respect default dag view on redirect after trigger > -- > > Key: AIRFLOW-7017 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7017 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Joshua Carp >Priority: Trivial > Fix For: 1.10.10 > > > Triggering a dag from the dag detail page always redirect to the dag tree > view. This redirect should respect the default dag view if configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] commented on issue #5499: [AIRFLOW-3964][AIP-17] Build smart sensor
stale[bot] commented on issue #5499: [AIRFLOW-3964][AIP-17] Build smart sensor URL: https://github.com/apache/airflow/pull/5499#issuecomment-606828635 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ShawonC opened a new issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0
ShawonC opened a new issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0 URL: https://github.com/apache/airflow/issues/8036 **Apache Airflow version**: 1.10.9 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.17.0 **Environment**: production - **Cloud provider or hardware configuration**: GKE - **OS** (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch) - **Kernel** (e.g. `uname -a`): Linux 0 4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT 2019 x86_64 GNU/Linux - **Install tools**: pip - **Others**: helm **What happened**: When running out salesforce dag, which uses a custom hook that wraps the base hook, the base hook gives us the following error: **What you expected to happen**: I expected the base SalesforceHook() class to properly sign in and pull data. It seems that this issue started occurring with the latest release of simple_salesforce, version 1.0.0. Based on the simple_salesforce code, it looks like the sign_in method in the SalesforceHook class in airflow/contrib/salesforce/hooks/salesforce_hook.py is still calling a deprecated keyword argument of `sandbox`. Here is a link to the commit for simple_salesforce 1.0.0 12 days ago: https://github.com/simple-salesforce/simple-salesforce/commit/1ebee0085b27ab5f9181a5e691fb8dc9fdc90163 **How to reproduce it**: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0
boring-cyborg[bot] commented on issue #8036: SalesforceHook not compatible with simple_salesforce 1.0.0 URL: https://github.com/apache/airflow/issues/8036#issuecomment-606819557 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7832: Add production image support
potiuk commented on issue #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#issuecomment-606808500 @ashb - are you OK with the explanations? I'd love to merge it and do some follow-ups (and cherry-pick it to 1.10.10). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-7017) Respect default dag view on redirect after trigger
[ https://issues.apache.org/jira/browse/AIRFLOW-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-7017: Fix Version/s: (was: 1.10.10) 2.0.0 > Respect default dag view on redirect after trigger > -- > > Key: AIRFLOW-7017 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7017 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Joshua Carp >Priority: Trivial > Fix For: 2.0.0 > > > Triggering a dag from the dag detail page always redirect to the dag tree > view. This redirect should respect the default dag view if configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7017) Respect default dag view on redirect after trigger
[ https://issues.apache.org/jira/browse/AIRFLOW-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-7017: Fix Version/s: (was: 2.0.0) 1.10.10 > Respect default dag view on redirect after trigger > -- > > Key: AIRFLOW-7017 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7017 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Joshua Carp >Priority: Trivial > Fix For: 1.10.10 > > > Triggering a dag from the dag detail page always redirect to the dag tree > view. This redirect should respect the default dag view if configured. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] konia-s opened a new issue #8035: Security: make user in mysql
konia-s opened a new issue #8035: Security: make user in mysql URL: https://github.com/apache/airflow/issues/8035 **Description** [reference](https://airflow.apache.org/docs/stable/security.html#web-authentication) I wanna make first user , so I ran this code. ``` $ cd ~/airflow $ python Python 2.7.9 (default, Feb 10 2015, 03:28:08) Type "help", "copyright", "credits" or "license" for more information. >>> import airflow >>> from airflow import models, settings >>> from airflow.contrib.auth.backends.password_auth import PasswordUser >>> user = PasswordUser(models.User()) >>> user.username = 'new_user_name' >>> user.email = 'new_user_em...@example.com' >>> user.password = 'set_the_password' >>> session = settings.Session() >>> session.add(user) >>> session.commit() >>> session.close() >>> exit() ``` Maybe this code is only sqlite ... How using mysql ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8035: Security: make user in mysql
boring-cyborg[bot] commented on issue #8035: Security: make user in mysql URL: https://github.com/apache/airflow/issues/8035#issuecomment-606785625 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too.
kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034#discussion_r401097531 ## File path: airflow/www/forms.py ## @@ -33,18 +33,20 @@ from airflow.utils import timezone from airflow.www.validators import ValidJson +from .widgets import AirflowDateTimePickerWidget Review comment: Yup This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8034: Make DateTimePicker actually pick times too.
ashb commented on a change in pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034#discussion_r401096659 ## File path: airflow/www/forms.py ## @@ -33,18 +33,20 @@ from airflow.utils import timezone from airflow.www.validators import ValidJson +from .widgets import AirflowDateTimePickerWidget Review comment: Yeah, the "explicit relative imports" bit is this. In this case it's not overly verbose without it so I guess for consistency i should use absolute. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too.
kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034#discussion_r401094796 ## File path: airflow/www/forms.py ## @@ -33,18 +33,20 @@ from airflow.utils import timezone from airflow.www.validators import ValidJson +from .widgets import AirflowDateTimePickerWidget Review comment: I am not sure either. But based on [PEP 8](https://www.python.org/dev/peps/pep-0008/): >Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path): >However, explicit relative imports are an acceptable alternative to absolute imports, especially when dealing with complex package layouts where using absolute imports would be unnecessarily verbose: So 路♂ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8034: Make DateTimePicker actually pick times too.
ashb commented on a change in pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034#discussion_r401092761 ## File path: airflow/www/forms.py ## @@ -33,18 +33,20 @@ from airflow.utils import timezone from airflow.www.validators import ValidJson +from .widgets import AirflowDateTimePickerWidget Review comment: I guess. I do wonder why we don't use relative imports more often... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too.
kaxil commented on a change in pull request #8034: Make DateTimePicker actually pick times too. URL: https://github.com/apache/airflow/pull/8034#discussion_r401092515 ## File path: airflow/www/forms.py ## @@ -33,18 +33,20 @@ from airflow.utils import timezone from airflow.www.validators import ValidJson +from .widgets import AirflowDateTimePickerWidget Review comment: ```suggestion from airflow.www.widgets import AirflowDateTimePickerWidget ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6837) limit description of dag in home page
[ https://issues.apache.org/jira/browse/AIRFLOW-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071990#comment-17071990 ] ASF subversion and git services commented on AIRFLOW-6837: -- Commit d15b6f5f9b7ae29d3ab83bf5d78a9134b3c46e2f in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d15b6f5 ] [AIRFLOW-6837] Limit description length of a Dag on HomePage (#7457) cherry-picked from 0ce2a55c7 > limit description of dag in home page > - > > Key: AIRFLOW-6837 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6837 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Fix For: 1.10.10 > > > so that the tooltip will not cover the dag link -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6730) is_alive uses seconds and not total_seconds
[ https://issues.apache.org/jira/browse/AIRFLOW-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071992#comment-17071992 ] ASF subversion and git services commented on AIRFLOW-6730: -- Commit 7ab04162c0c3c87539a32f41f367d78a75eeef53 in airflow's branch refs/heads/v1-10-test from Alex Guziel [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7ab0416 ] [AIRFLOW-6730] Use total_seconds instead of seconds (#7363) * [AIRFLOW-6730] Use total_seconds instead of seconds * adds tests and fixes types issue * fix test (cherry picked from commit 008b4bab14222da068b737d6332db4963b994007) > is_alive uses seconds and not total_seconds > --- > > Key: AIRFLOW-6730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6730 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.4 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0, 1.10.10 > > > Example: > timedelta(days=1).seconds = 0 > timedelta(days=1).total_seconds() = 86400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6695) Support passing DAG configuration in UI
[ https://issues.apache.org/jira/browse/AIRFLOW-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071986#comment-17071986 ] ASF subversion and git services commented on AIRFLOW-6695: -- Commit 620a9e9169d3861274515d2e6a215c454630e593 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=620a9e9 ] [AIRFLOW-6695] Can now pass dagrun conf when triggering dags via UI (#7312) cherry-picked from f0c31c597 > Support passing DAG configuration in UI > --- > > Key: AIRFLOW-6695 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6695 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.10 >Reporter: Daniel Cohen >Priority: Minor > Fix For: 1.10.10 > > > When triggering a DAG from the CLI or the REST API, it's possible to pass > configuration for the DAG run as a JSON blob. > The PR accompanying this ticket adds a new screen confirming the trigger > request, and allowing the user to pass a JSON configuration blob to the new > DAG run. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6628) Dag search autocomplete is filtered by page context
[ https://issues.apache.org/jira/browse/AIRFLOW-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071989#comment-17071989 ] ASF subversion and git services commented on AIRFLOW-6628: -- Commit eae2394189fd09e34e522194f9e5842437445ac3 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=eae2394 ] [AIRFLOW-6628] DAG auto-complete now suggests from all acessible DAGs (#7251) The auto complete on the dag search box utilises the page search context which prevents auto completing dags which dont match the current query. On the dags page with no search provided it executes a query that loads every dag_id in the system to pass to the typeahead widget. On systems with a lot of dags this makes the page large and slow. Add a JSON endpoint to provide correct auto complete behaviour and reduce page load time. cherry-picked from 1a9d21861 > Dag search autocomplete is filtered by page context > --- > > Key: AIRFLOW-6628 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6628 > Project: Apache Airflow > Issue Type: Bug > Components: ui, webserver >Affects Versions: 2.0.0, 1.10.7, 1.10.10 >Reporter: Robin Edwards >Assignee: Robin Edwards >Priority: Minor > Fix For: 1.10.10 > > > The auto complete on the dag search box utilises the page search context > which prevents it auto completing dags which dont match the current query > Further still on the home page with no search provided it executes a query > that loads every dag_id in the system for it to be passed to the typeahead > widget. On systems with a lot of dags this makes the page slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5590) Add 'run_id' to trigger DAG run API response as a field
[
https://issues.apache.org/jira/browse/AIRFLOW-5590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071987#comment-17071987
]
ASF subversion and git services commented on AIRFLOW-5590:
--
Commit e840ce8242a80e5e80caf88e00cc597d738bfbb5 in airflow's branch
refs/heads/v1-10-test from Kumpan Anton
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e840ce8 ]
[AIRFLOW-5590] Add run_id to trigger DAG run API response (#6256)
> Add 'run_id' to trigger DAG run API response as a field
> ---
>
> Key: AIRFLOW-5590
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5590
> Project: Apache Airflow
> Issue Type: Improvement
> Components: api
>Affects Versions: 1.10.5
>Reporter: Anton Kumpan
>Assignee: Anton Kumpan
>Priority: Major
> Fix For: 1.10.10
>
>
> Idea is the same as in AIRFLOW-4482
> Currently API response looks like this:
> {code:json}
> {
> 'execution_date': '2019-05-08T07:03:09+00:00',
> 'message': 'Created manual__2019-05-08T07:03:09+00:00, externally triggered: True>'
> }
> {code}
>
> It would be nice to add run_id as a separate field, so that response will
> look like:
> {code:json}
> {
> 'run_id': 'manual__2019-05-08T07:03:09+00:00',
> 'execution_date': '2019-05-08T07:03:09+00:00',
> 'message': 'Created triggered: True>'
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7080) API Endpoint to query a dag's paused status
[ https://issues.apache.org/jira/browse/AIRFLOW-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071988#comment-17071988 ] ASF subversion and git services commented on AIRFLOW-7080: -- Commit 933abafa8f07660208629b966bacbdaea1bf1412 in airflow's branch refs/heads/v1-10-test from Philipp Großelfinger [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=933abaf ] [AIRFLOW-7080] Adds API endpoint to return a DAG's paused state (#7737) Adds an additional endpoint to the experimental API to return the paused state of a DAG. > API Endpoint to query a dag's paused status > --- > > Key: AIRFLOW-7080 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7080 > Project: Apache Airflow > Issue Type: New Feature > Components: api >Affects Versions: 2.0.0, 1.10.10 >Reporter: Philipp Großelfinger >Assignee: Philipp Großelfinger >Priority: Minor > Fix For: 1.10.10 > > > So far it is possible to set the paused state of a DAG via the experimental > API. It would be nice to be also able to query the current paused state of a > DAG. > The endpoint to set the paused state looks like this: > /api/experimental/dags//paused/ > The new endpoint could look like: > GET /api/experimental/dags//paused -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6167) Escape col name in MysqlToHive operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071991#comment-17071991 ] ASF subversion and git services commented on AIRFLOW-6167: -- Commit b1ab39f548745bf293bf7f5ba306f8ed85aab92e in airflow's branch refs/heads/v1-10-test from Ping Zhang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b1ab39f ] [AIRFLOW-6167] Escape column name in create table in hive (#6741) Hive query will fail if column name contains keyword, unless escaped. Fix is always escaping column name when creating table. (cherry picked from commit a1f3b29344d400aefd4a6c1f6842d2a345de) > Escape col name in MysqlToHive operator > --- > > Key: AIRFLOW-6167 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6167 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.4 >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Fix For: 1.10.10 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #8021: Allow setting Airflow Variable values to empty string ('')
kaxil merged pull request #8021: Allow setting Airflow Variable values to empty
string ('')
URL: https://github.com/apache/airflow/pull/8021
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] RosterIn commented on issue #8021: Allow setting Airflow Variable values to empty string ('')
RosterIn commented on issue #8021: Allow setting Airflow Variable values to
empty string ('')
URL: https://github.com/apache/airflow/pull/8021#issuecomment-606755738
Why does it need to be mentioned in updating.md of master if this was
already release in 1.10.5?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on issue #8029: Update TESTING.rst
kaxil commented on issue #8029: Update TESTING.rst URL: https://github.com/apache/airflow/pull/8029#issuecomment-606750151 @yajna-pandith Can you add more descriptive PR title please for other PRs? This title/commit message would appear in CHANGELOG and it isn't super clear from "Updating README.MD" on what actually was updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #8028: Update BREEZE.rst
kaxil commented on issue #8028: Update BREEZE.rst URL: https://github.com/apache/airflow/pull/8028#issuecomment-606750125 @yajna-pandith Can you add more descriptive PR title please for other PRs? This title/commit message would appear in CHANGELOG and it isn't super clear from "Updating README.MD" on what actually was updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7895: Upgrade flask-appbuilder to ~=2.3
kaxil commented on issue #7895: Upgrade flask-appbuilder to ~=2.3 URL: https://github.com/apache/airflow/pull/7895#issuecomment-606749323 @feluelle Can you resolve conflicts please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #8027: Updating README.MD
kaxil commented on issue #8027: Updating README.MD URL: https://github.com/apache/airflow/pull/8027#issuecomment-606748667 @yajna-pandith Can you add more descriptive PR title please for other PRs? This title/commit message would appear in CHANGELOG and it isn't super clear from "Updating README.MD" on what actually was updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6562) mushroom cloud error when clicking 'mark failed/success' from graph view of dag that has never been run yet
[
https://issues.apache.org/jira/browse/AIRFLOW-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-6562:
Fix Version/s: (was: 1.10.10)
2.0.0
> mushroom cloud error when clicking 'mark failed/success' from graph view of
> dag that has never been run yet
> ---
>
> Key: AIRFLOW-6562
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6562
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.6
> Environment: localexec, mysql metastore, 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 2.0.0
>
>
> # create a new dag
> # go to graph view
> # click on one of the tasks (it should have a white border)
> # click on 'past/future' on either 2nd last row (mark failed) or last row
> (mark success)
> # then click either (mark failed) or (mark success)
> below error appears
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in
> reraise
> raise value
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 69,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/flask_admin/base.py", line 368,
> in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line
> 258, in decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line
> 290, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line
> 337, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line
> 1449, in failed
> future, past, State.FAILED)
> File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line
> 1420, in _mark_task_instance_state
> commit=False)
> File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74,
> in wrapper
> return func(*args, **kwargs)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
> line 105, in set_state
> dates = get_execution_dates(dag, execution_date, future, past)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/mark_tasks.py",
> line 246, in get_execution_dates
> raise ValueError("Received non-localized date {}".format(execution_date))
> ValueError: Received non-localized date 2020-01-14T21:58:44.855743+00:00
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6994) SparkSubmitOperator re launches spark driver even when original driver still running
[
https://issues.apache.org/jira/browse/AIRFLOW-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik updated AIRFLOW-6994:
Fix Version/s: (was: 1.10.10)
2.0.0
> SparkSubmitOperator re launches spark driver even when original driver still
> running
>
>
> Key: AIRFLOW-6994
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6994
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.8, 1.10.9
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/AIRFLOW-6229 introduced a bug
> Due to temporary network blip in connection to spark the state goes to
> unknown (as no tags found in curl response) and forces retry
> fix in spark_submit_hook.py:
>
> {code:java}
> def _process_spark_status_log(self, itr):
> """
> parses the logs of the spark driver status query process
> :param itr: An iterator which iterates over the input of the
> subprocess
> """
> response_found = False
> driver_found = False
> # Consume the iterator
> for line in itr:
> line = line.strip()
> if "submissionId" in line:
> response_found = True
>
> # Check if the log line is about the driver status and extract
> the status.
> if "driverState" in line:
> self._driver_status = line.split(' : ')[1] \
> .replace(',', '').replace('\"', '').strip()
> driver_found = True
> self.log.debug("spark driver status log: {}".format(line))
> if response_found and not driver_found:
> self._driver_status = "UNKNOWN"
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7832: Add production image support
potiuk commented on issue #7832: Add production image support URL: https://github.com/apache/airflow/pull/7832#issuecomment-606709575 All corrections made! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
