[GitHub] [airflow] dstandish opened a new pull request #8257: [docs-only] Simplify language re roll-your-own secrets backend
dstandish opened a new pull request #8257: [docs-only] Simplify language re roll-your-own secrets backend URL: https://github.com/apache/airflow/pull/8257 * remove the "simply" and "just" modifiers -- too infomercialey * simplify language surrounding "options" -- i.e. whether to implement get_conn_uri or get_connections --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS.
codecov-io edited a comment on issue #7128: [AIRFLOW-6529] Pickle error occurs when the scheduler tries to run on macOS. URL: https://github.com/apache/airflow/pull/7128#issuecomment-573300577 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=h1) Report > Merging [#7128](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/30b54d3ecbeb2ebe6c7fcb2cae9f00210a41e6f7=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `89.65%`. [](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7128 +/- ## == - Coverage 85.40% 85.08% -0.33% == Files 710 723 +13 Lines 3948539564 +79 == - Hits3372433663 -61 - Misses 5761 5901 +140 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.16% <84.61%> (-0.19%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.24% <93.75%> (+0.05%)` | :arrow_up: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0.00% <0.00%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0.00% <0.00%> (-81.25%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0.00%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0.00%> (-20.49%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.01% <0.00%> (-3.43%)` | :arrow_down: | | ... and [79 more](https://codecov.io/gh/apache/airflow/pull/7128/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=footer). Last update [30b54d3...b612321](https://codecov.io/gh/apache/airflow/pull/7128?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dmmakita commented on issue #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor
dmmakita commented on issue #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor URL: https://github.com/apache/airflow/pull/7407#issuecomment-612543866 Hi, @dferguson992, I'm having similar needs related to airflow-kafka. May I try to help you with this issue #7407? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7267: [AIRFLOW-6645] Pass the TaskHandlers configuration using the constructor
stale[bot] commented on issue #7267: [AIRFLOW-6645] Pass the TaskHandlers configuration using the constructor URL: https://github.com/apache/airflow/pull/7267#issuecomment-612541166 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on issue #8145: Drop redundant project id null checks.
jmcarp commented on issue #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#issuecomment-612530509 Thanks for reviewing. Is it all right to merge while `TestCliTaskBackfill.test_run_ignores_all_dependencies` is failing? It looks unrelated to this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH edited a comment on issue #8255: Config value interpolation not working when set as environment variable
BasPH edited a comment on issue #8255: Config value interpolation not working when set as environment variable URL: https://github.com/apache/airflow/issues/8255#issuecomment-612505957 Dug a bit more into this issue. It seems Airflow builds its configuration in a rather odd way. First there is an `AirflowConfigParser`, which is the class you get when importing `from airflow.configuration import conf`. In addition to overriding e.g. `ConfigParser.get()`, to support fetching configuration from environment variables and others, the AirflowConfigParser also holds an attribute `airflow_defaults` with is a "standard" ConfigParser by itself, used for looking up default values if none of the custom ways for setting config are used. This makes it difficult to fix this issue. Interpolation of config values is done in an `Interpolation` class, which can be overridden to also support fetching values from the custom config. However, since there a (standard) ConfigParser inside the AirflowConfigParser for the default values, we'd have to hack around to also support the custom config fetching ways in there. All in all, it might be technically possible to fix, but I'd rather opt for rewriting the whole configuration module, to avoid more hacking and take the opportunity to create something simple to understand. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on issue #8255: Config value interpolation not working when set as environment variable
BasPH commented on issue #8255: Config value interpolation not working when set as environment variable URL: https://github.com/apache/airflow/issues/8255#issuecomment-612505957 Dug a bit more into this issue. It seems Airflow builds its configuration in a rather odd way. First there is an `AirflowConfigParser`, which is the class you get when importing `from airflow.configuration import conf`. In addition to overriding e.g. `ConfigParser.get()`, to support fetching configuration from environment variables and others, the AirflowConfigParser also holds an attribute `airflow_defaults` with is a "standard" ConfigParser by itself, used for looking up default values if none of the custom ways for setting config are used. This makes it difficult to fix this issue. Interpolation of config values is done in an `Interpolation` class, which can be overridden to also support fetching values from the custom config. However, since there a (standard) ConfigParser inside the AirflowConfigParser for the default values, we'd have to hack around to also support the custom config fetching ways in there. All in all, it might be technically possible to fix, but I'd rather opt for rewriting the whole configuration module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] maganaluis commented on issue #8177: provide_context=True not working with PythonVirtualenvOperator
maganaluis commented on issue #8177: provide_context=True not working with PythonVirtualenvOperator URL: https://github.com/apache/airflow/issues/8177#issuecomment-612496365 @mik-laj Hi Kamil - I created the PR #8256 to fix this issue on the branch v1-10-stable and the CI tests are passing, there seems to be an issue with requirements, but I that's related to this change. Could you let me know the next steps? There were too many changes on the master branch, I will revisit this bug there once 2.0 is out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] maganaluis opened a new pull request #8256: updated _write_args on PythonVirtualenvOperator
maganaluis opened a new pull request #8256: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8256 This change will ensure that argument provide_context=True works on v1.10.10 for PythonVirtualenvOperator Related bug: https://github.com/apache/airflow/issues/8177 --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] maganaluis closed pull request #8224: updated _write_args on PythonVirtualenvOperator
maganaluis closed pull request #8224: updated _write_args on PythonVirtualenvOperator URL: https://github.com/apache/airflow/pull/8224 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] khyurri edited a comment on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg
khyurri edited a comment on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg URL: https://github.com/apache/airflow/issues/8237#issuecomment-612467387 Bug confirmed at 1.10.10 with celery executor. If i've got time tomorrow, I'll try to explore it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] khyurri edited a comment on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg
khyurri edited a comment on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg URL: https://github.com/apache/airflow/issues/8237#issuecomment-612467387 Bug confirmed at 1.10.10 with celery executor. If i've got time tomorrow, I'll try to explore it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] khyurri commented on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg
khyurri commented on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg URL: https://github.com/apache/airflow/issues/8237#issuecomment-612467387 Bug confirmed at 1.10.10 with celery executor. If i've got time tomorrow, I'll try to explore it tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH opened a new issue #8255: Config value interpolation not working when set as environment variable
BasPH opened a new issue #8255: Config value interpolation not working when set as environment variable URL: https://github.com/apache/airflow/issues/8255 **Apache Airflow version**: 1.10.10 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: I set store_serialized_dags as an environment variable: `export AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True`. The default value of `store_dag_code` is `%(store_serialized_dags)s`, so I expect it to now also be True. However, it's not (it's False). **What you expected to happen**: I expect store_dag_code to have the same value as store_serialized_dags. I think it's because the interpolation is applied to the value set in .cfg, but not when it's set via an env var. Will have to double check. **How to reproduce it**: **Anything else we need to know**: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Nakulbajaj101 opened a new pull request #8254: changed sendgrid apikey parameter to api_key
Nakulbajaj101 opened a new pull request #8254: changed sendgrid apikey parameter to api_key URL: https://github.com/apache/airflow/pull/8254 fixed the error TypeError: __init__() got an unexpected keyword argument 'apikey', should be api_key --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8254: changed sendgrid apikey parameter to api_key
boring-cyborg[bot] commented on issue #8254: changed sendgrid apikey parameter to api_key URL: https://github.com/apache/airflow/pull/8254#issuecomment-612462506 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
BasPH commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253#issuecomment-612460619 Thanks @KostyaEsmukov! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH merged pull request #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
BasPH merged pull request #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
boring-cyborg[bot] commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253#issuecomment-612460706 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
codecov-io edited a comment on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253#issuecomment-612459830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=h1) Report > Merging [#8253](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/eb17ef04b04b220b31313da9e50f6485af1e0ab2=desc) will **decrease** coverage by `55.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#8253 +/- ## === - Coverage 88.32% 33.23% -55.09% === Files 936 936 Lines 4531945319 === - Hits4003015064-24966 - Misses 528930255+24966 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [807 more](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=footer). Last update [eb17ef0...0caf1df](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #8251: Add documentation for CLI command airflow dags test
potiuk merged pull request #8251: Add documentation for CLI command airflow dags test URL: https://github.com/apache/airflow/pull/8251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
codecov-io edited a comment on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253#issuecomment-612459830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=h1) Report > Merging [#8253](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/eb17ef04b04b220b31313da9e50f6485af1e0ab2=desc) will **decrease** coverage by `55.11%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#8253 +/- ## === - Coverage 88.32% 33.21% -55.12% === Files 936 936 Lines 4531945319 === - Hits4003015051-24979 - Misses 528930268+24979 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [808 more](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=footer). Last update [eb17ef0...0caf1df](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
codecov-io commented on issue #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253#issuecomment-612459830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=h1) Report > Merging [#8253](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/eb17ef04b04b220b31313da9e50f6485af1e0ab2=desc) will **decrease** coverage by `55.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#8253 +/- ## === - Coverage 88.32% 33.20% -55.13% === Files 936 936 Lines 4531945319 === - Hits4003015046-24984 - Misses 528930273+24984 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | [airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: | | ... and [808 more](https://codecov.io/gh/apache/airflow/pull/8253/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=footer). Last update [eb17ef0...0caf1df](https://codecov.io/gh/apache/airflow/pull/8253?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407081546 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: I think we do not need to get them auto-generated in the prod version. The main reason they are auto-generated is that I wanted to keep the the same in docker-compose and in the scripts which are running pre-commit checks. Pre-commit checks do not use docker compose but they are running docker run commands directly. This only makes sense for CI image. For prod image it's ok to keep separate list I think - most of the pre-commit/testrelated files might be misleading/are unnecessary in the production image. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__
kaxil edited a comment on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__ URL: https://github.com/apache/airflow/pull/8225#issuecomment-612452255 Thanks for confirming @KostyaEsmukov Yes this will be backported to v1-10 and would be available in 1.10.11 :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KostyaEsmukov commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__
KostyaEsmukov commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__ URL: https://github.com/apache/airflow/pull/8225#issuecomment-612452509 Great, thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil edited a comment on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__
kaxil edited a comment on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__ URL: https://github.com/apache/airflow/pull/8225#issuecomment-612452255 Thanks for confirming. Yes this will be backported to v1-10 and would be available in 1.10.11 :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__
kaxil commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__ URL: https://github.com/apache/airflow/pull/8225#issuecomment-612452255 Yes this will be backported to v1-10 and would be available in 1.10.11 :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KostyaEsmukov commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__
KostyaEsmukov commented on issue #8225: Move DAG._schedule_interval logic out of DAG.__init__ URL: https://github.com/apache/airflow/pull/8225#issuecomment-612451967 @kaxil thank you for this fix! I have backported it to 1.10 and applied on my installation (instead of a patch in https://github.com/apache/airflow/issues/8166#issuecomment-610005896) and can confirm that it indeed fixes the issue. Will this be backported to v1-10? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KostyaEsmukov opened a new pull request #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares
KostyaEsmukov opened a new pull request #8253: RBAC ui: fix missing task runs being rendered as circles instead of squares URL: https://github.com/apache/airflow/pull/8253 In RBAC ui there's a bug: when an existing DAG is expanded with new tasks, the missing taskinstances for previous DAG runs are being rendered as circles (like dagruns) instead of squares: https://user-images.githubusercontent.com/2418596/79048074-35691480-7c23-11ea-85c5-3886b24426f6.png;> This PR fixes that, making it look like this (as it was in the legacy flask-admin UI): https://user-images.githubusercontent.com/2418596/79048086-4154d680-7c23-11ea-9075-dd0cc76689a7.png;> The check now looks exactly as it is in the flask-admin UI: https://github.com/apache/airflow/blob/1.10.10/airflow/www/templates/airflow/tree.html#L249 In RBAC for some reason it is different: https://github.com/apache/airflow/blob/1.10.10/airflow/www_rbac/templates/airflow/tree.html#L91 I'm not sure if the change was intentional, and also I'm not sure if the fix is proper, but it seems to work that way. --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] ~~Unit tests coverage for changes (not needed for documentation changes)~~ - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] ~~Relevant documentation is updated including usage instructions.~~ - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KostyaEsmukov commented on issue #8252: RBAC ui: fix missing Y-axis labels with units in plots
KostyaEsmukov commented on issue #8252: RBAC ui: fix missing Y-axis labels with units in plots URL: https://github.com/apache/airflow/pull/8252#issuecomment-612447155 Tested on master with Breeze. master branch (no units): https://user-images.githubusercontent.com/2418596/79047926-52511800-7c22-11ea-9b1c-f22407fe684d.png;> This branch (units are visible): https://user-images.githubusercontent.com/2418596/79047921-4f562780-7c22-11ea-9103-2e3ab8b4e239.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8252: RBAC ui: fix missing Y-axis labels with units in plots
boring-cyborg[bot] commented on issue #8252: RBAC ui: fix missing Y-axis labels with units in plots URL: https://github.com/apache/airflow/pull/8252#issuecomment-612442999 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KostyaEsmukov opened a new pull request #8252: RBAC ui: fix missing Y-axis labels with units in plots
KostyaEsmukov opened a new pull request #8252: RBAC ui: fix missing Y-axis labels with units in plots URL: https://github.com/apache/airflow/pull/8252 In RBAC ui there's a bug in the Task Duration and Landing Times plots: Y-axis label (with the time units) is not visible: https://user-images.githubusercontent.com/2418596/79047325-92160080-7c1e-11ea-88ea-ebf18f303a7d.png;> In the legacy flask-admin UI there's no such issue (don't mind the different data on the plots, these are two different installations): https://user-images.githubusercontent.com/2418596/79047329-96dab480-7c1e-11ea-81c5-9986f82e525f.png;> (Notice that the `y`-offset in the object inspector is different). This PR fixes the issue by changing the offset, making it the same as it was in the legacy UI: https://user-images.githubusercontent.com/2418596/79047327-95a98780-7c1e-11ea-917b-27dba08c38f4.png;> I suppose the reason why this broke is that in flask-admin the `d3` js library is vendored and has not been updated since 2016: https://github.com/apache/airflow/commits/v1-10-test/airflow/www/static/d3.v3.min.js , but in RBAC a newer version of d3 is installed via package.json. Perhaps the interpretation of `axisLabelDistance` has changed in the newer versions of d3? I have tested this patch on Airflow 1.10.10. Didn't test on master yet, but I'm going to if I manage to get along with the breeze environment. --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [ ] ~~Unit tests coverage for changes (not needed for documentation changes)~~ - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] ~~Relevant documentation is updated including usage instructions.~~ - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8213: Fix issue #8163 where gcp auth fails
potiuk commented on issue #8213: Fix issue #8163 where gcp auth fails URL: https://github.com/apache/airflow/pull/8213#issuecomment-612442519 Travis has database maintenance today: https://www.traviscistatus.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] iamshwin commented on issue #8213: Fix issue #8163 where gcp auth fails
iamshwin commented on issue #8213: Fix issue #8163 where gcp auth fails URL: https://github.com/apache/airflow/pull/8213#issuecomment-612429788 I can't see any logs in Travis? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407065268 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: or maybe we keep it for now.. because these get auto-generated and I am not really sure what to change in the _utils for it to work. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407064781 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: looks good. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result
stale[bot] commented on issue #7508: [AIRFLOW-6886] Cleanup the SageMakerTrainingOperator result URL: https://github.com/apache/airflow/pull/7508#issuecomment-612418774 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command URL: https://github.com/apache/airflow/pull/7426#issuecomment-612414301 PR: https://github.com/apache/airflow/pull/8251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH opened a new pull request #8251: Add documentation for CLI command airflow dags test
BasPH opened a new pull request #8251: Add documentation for CLI command airflow dags test URL: https://github.com/apache/airflow/pull/8251 This PR adds a paragraph to the docs explaining `airflow dags test`, as requested https://github.com/apache/airflow/pull/7426#issuecomment-601653089. There are tests now so I believe that part is resolved. --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #8250: Fix indentation in GitHub bug report template
kaxil commented on issue #8250: Fix indentation in GitHub bug report template URL: https://github.com/apache/airflow/pull/8250#issuecomment-612412002 arghh, it took the commit title instead of PR as it had just 1 commit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #8250: Fix indentation in GitHub bug report template
kaxil merged pull request #8250: Fix indentation in GitHub bug report template URL: https://github.com/apache/airflow/pull/8250 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH opened a new pull request #8250: Fix indentation
BasPH opened a new pull request #8250: Fix indentation URL: https://github.com/apache/airflow/pull/8250 This fixes a small nit in the GitHub bug report template. Before:  After:  --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #8145: Drop redundant project id null checks.
turbaszek commented on issue #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#issuecomment-612383137 I still have concerns about type annotations consistency: https://github.com/apache/airflow/pull/8145#discussion_r407044132 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8145: Drop redundant project id null checks.
turbaszek commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407044132 ## File path: airflow/providers/google/cloud/hooks/bigtable.py ## @@ -54,7 +54,7 @@ def _get_client(self, project_id: str): return self._client @GoogleBaseHook.fallback_to_default_project_id -def get_instance(self, instance_id: str, project_id: Optional[str] = None) -> Instance: +def get_instance(self, instance_id: str, project_id: str) -> Instance: Review comment: @jmcarp I checked the changes and in some places where we use the decorator we are using `project_id: str` and in others `project_id: Optional[str] = None`. I think we should be consistent :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407043149 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: Well some of those might be useful as they give a lot of nice things when it comes to development. They keep history of commands, you can define your own aliases etc + you can for example build backport packages outside (in dist) and install them inside the container. Some of the stuff is used for pylint/mypy/etc so indeed it is not relevant: I reviewed the list and I think this - shorter - list might make sense to stay: ``` - ../../../.bash_aliases:/root/.bash_aliases:cached - ../../../.bash_history:/root/.bash_history:cached - ../../../.github:/opt/airflow/.github:cached - ../../../.inputrc:/root/.inputrc:cached - ../../../.kube:/root/.kube:cached - ../../../airflow:/opt/airflow/airflow:cached - ../../../common:/opt/airflow/common:cached - ../../../dags:/opt/airflow/dags:cached - ../../../files:/files:cached - ../../../dist:/dist:cached - ../../../requirements:/opt/airflow/requirements:cached - ../../../scripts:/opt/airflow/scripts:cached - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached - ../../../setup.cfg:/opt/airflow/setup.cfg:cached - ../../../setup.py:/opt/airflow/setup.py:cached - ../../../tests:/opt/airflow/tests:cached - ../../../tmp:/opt/airflow/tmp:cached ``` And remember we can skip it via `--skip-mounting-local-sources` flag to breeze as well - to have a vanilla production container without any mounts.
[GitHub] [airflow] feluelle merged pull request #8229: Proper version is displayed when running prod image via Breeze
feluelle merged pull request #8229: Proper version is displayed when running prod image via Breeze URL: https://github.com/apache/airflow/pull/8229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407041384 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: the linting, ci and testing stuff.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407041243 ## File path: scripts/ci/docker-compose/local-prod.yml ## @@ -0,0 +1,66 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +version: "2.2" +services: + airflow: +# We need to mount files an directories individually because some files +# such apache_airflow.egg-info should not be mounted from host +# we only mount those files that it makes sense to edit while developing +# or those that might be useful to see in the host as output of the +# tests (such as logs) +volumes: + # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh + - ../../../.bash_aliases:/root/.bash_aliases:cached + - ../../../.bash_history:/root/.bash_history:cached + - ../../../.coveragerc:/opt/airflow/.coveragerc:cached + - ../../../.dockerignore:/opt/airflow/.dockerignore:cached + - ../../../.flake8:/opt/airflow/.flake8:cached + - ../../../.github:/opt/airflow/.github:cached + - ../../../.inputrc:/root/.inputrc:cached + - ../../../.kube:/root/.kube:cached + - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached + - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached + - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached + - ../../../LICENSE:/opt/airflow/LICENSE:cached + - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached + - ../../../NOTICE:/opt/airflow/NOTICE:cached + - ../../../airflow:/opt/airflow/airflow:cached + - ../../../common:/opt/airflow/common:cached + - ../../../dags:/opt/airflow/dags:cached + - ../../../dev:/opt/airflow/dev:cached + - ../../../docs:/opt/airflow/docs:cached + - ../../../files:/files:cached + - ../../../dist:/dist:cached + - ../../../hooks:/opt/airflow/hooks:cached + - ../../../logs:/root/airflow/logs:cached + - ../../../pylintrc:/opt/airflow/pylintrc:cached + - ../../../pytest.ini:/opt/airflow/pytest.ini:cached + - ../../../requirements:/opt/airflow/requirements:cached + - ../../../scripts:/opt/airflow/scripts:cached + - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached + - ../../../setup.cfg:/opt/airflow/setup.cfg:cached + - ../../../setup.py:/opt/airflow/setup.py:cached + - ../../../tests:/opt/airflow/tests:cached + - ../../../tmp:/opt/airflow/tmp:cached + # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh Review comment: We don't need any of those volumes, do we? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image URL: https://github.com/apache/airflow/pull/8228#discussion_r407041120 ## File path: scripts/ci/docker-compose/backend-mysql.yml ## @@ -25,6 +25,8 @@ services: - AIRFLOW__CELERY__RESULT_BACKEND=db+mysql://root@mysql/airflow?charset=utf8mb4 depends_on: - mysql +ports: Review comment: Like this https://github.com/apache/airflow/pull/8228/files ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8145: Drop redundant project id null checks.
codecov-io commented on issue #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#issuecomment-612370458 # [Codecov](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=h1) Report > Merging [#8145](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/12b9b64e10626bb1bc382815edeac9cbedd2701a=desc) will **decrease** coverage by `0.23%`. > The diff coverage is `27.65%`. [](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=tree) ```diff @@Coverage Diff @@ ## master#8145 +/- ## == - Coverage 88.32% 88.08% -0.24% == Files 936 937 +1 Lines 4531945195 -124 == - Hits4002939812 -217 - Misses 5290 5383 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/mypy/plugin/decorators.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9teXB5L3BsdWdpbi9kZWNvcmF0b3JzLnB5) | `0.00% <0.00%> (ø)` | | | [...google/cloud/example\_dags/example\_automl\_tables.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2F1dG9tbF90YWJsZXMucHk=) | `92.72% <ø> (ø)` | | | [airflow/providers/google/cloud/hooks/automl.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2F1dG9tbC5weQ==) | `98.94% <ø> (+10.42%)` | :arrow_up: | | [...rflow/providers/google/cloud/hooks/bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2JpZ3F1ZXJ5X2R0cy5weQ==) | `86.00% <ø> (+6.68%)` | :arrow_up: | | [...irflow/providers/google/cloud/hooks/cloud\_build.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2Nsb3VkX2J1aWxkLnB5) | `100.00% <ø> (+2.56%)` | :arrow_up: | | [airflow/providers/google/cloud/hooks/cloud\_sql.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2Nsb3VkX3NxbC5weQ==) | `67.58% <ø> (+0.33%)` | :arrow_up: | | [...ogle/cloud/hooks/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2Nsb3VkX3N0b3JhZ2VfdHJhbnNmZXJfc2VydmljZS5weQ==) | `89.67% <ø> (+0.99%)` | :arrow_up: | | [...irflow/providers/google/cloud/hooks/datacatalog.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2RhdGFjYXRhbG9nLnB5) | `91.74% <ø> (ø)` | | | [airflow/providers/google/cloud/hooks/datafusion.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2RhdGFmdXNpb24ucHk=) | `84.53% <ø> (+3.22%)` | :arrow_up: | | [airflow/providers/google/cloud/hooks/mlengine.py](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL21sZW5naW5lLnB5) | `82.74% <ø> (+1.27%)` | :arrow_up: | | ... and [23 more](https://codecov.io/gh/apache/airflow/pull/8145/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=footer). Last update [12b9b64...e0f61d7](https://codecov.io/gh/apache/airflow/pull/8145?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8215: Fix breeze building production image not recognizing EXTRA_DOCKER_PROD_BUILD_FLAGS
feluelle commented on issue #8215: Fix breeze building production image not recognizing EXTRA_DOCKER_PROD_BUILD_FLAGS URL: https://github.com/apache/airflow/pull/8215#issuecomment-612369362 Thanks @potiuk - can confirm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle closed pull request #8215: Fix breeze building production image not recognizing EXTRA_DOCKER_PROD_BUILD_FLAGS
feluelle closed pull request #8215: Fix breeze building production image not recognizing EXTRA_DOCKER_PROD_BUILD_FLAGS URL: https://github.com/apache/airflow/pull/8215 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH commented on issue #8233: Dag tags use set instead of list
BasPH commented on issue #8233: Dag tags use set instead of list URL: https://github.com/apache/airflow/pull/8233#issuecomment-612366458 @kaxil I think it might be worth changing this call to a set: https://github.com/apache/airflow/blob/master/airflow/models/dag.py#L1532. It seems currently the DagTags are made unique when fetching from the DB. Other than that, we can close this pull request IMO. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JPFrancoia commented on issue #7905: Pandas version pinned to < 1.0.0
JPFrancoia commented on issue #7905: Pandas version pinned to < 1.0.0 URL: https://github.com/apache/airflow/issues/7905#issuecomment-612366286 Closing since PR was merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JPFrancoia closed issue #7905: Pandas version pinned to < 1.0.0
JPFrancoia closed issue #7905: Pandas version pinned to < 1.0.0 URL: https://github.com/apache/airflow/issues/7905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407035131 ## File path: airflow/providers/google/cloud/example_dags/example_automl_tables.py ## @@ -139,7 +139,6 @@ def get_target_column_spec(columns_specs: List[Dict], column_name: str) -> str: task_id="update_dataset_task", dataset=update, location=GCP_AUTOML_LOCATION, -project_id=GCP_PROJECT_ID, Review comment: Yeah. I thik it's good :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8145: Drop redundant project id null checks.
potiuk commented on issue #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#issuecomment-612365354 @turbaszek @mik-laj -> looks good to me. Can you take another look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id
null checks.
URL: https://github.com/apache/airflow/pull/8145#discussion_r407034896
##
File path: tests/providers/google/cloud/hooks/test_cloud_sql.py
##
@@ -561,29 +513,6 @@ def test_get_instance_overridden_project_id(
execute_method.assert_called_once_with(num_retries=5)
wait_for_operation_to_complete.assert_not_called()
-@mock.patch(
-
'airflow.providers.google.common.hooks.base_google.GoogleBaseHook.project_id',
-new_callable=PropertyMock,
-return_value=None
-)
-
@mock.patch('airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook.get_conn')
-
@mock.patch('airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook._wait_for_operation_to_complete')
-def test_get_instance_missing_project_id(
-self, wait_for_operation_to_complete, get_conn, mock_project_id
-):
-get_method = get_conn.return_value.instances.return_value.get
-execute_method = get_method.return_value.execute
-execute_method.return_value = {"name": "instance"}
-wait_for_operation_to_complete.return_value = None
-with self.assertRaises(AirflowException) as cm:
-self.cloudsql_hook_no_default_project_id.get_instance(
-instance='instance')
-get_method.assert_not_called()
-execute_method.assert_not_called()
-err = cm.exception
-self.assertIn("The project id must be passed", str(err))
-wait_for_operation_to_complete.assert_not_called()
-
Review comment:
Agree. No need to repeat it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS Glue Job Integration
feluelle commented on a change in pull request #6007: [AIRFLOW-2310] Enable AWS
Glue Job Integration
URL: https://github.com/apache/airflow/pull/6007#discussion_r407034889
##
File path: airflow/providers/amazon/aws/operators/glue.py
##
@@ -0,0 +1,117 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import unicode_literals
+
+from airflow.providers.amazon.aws.hooks.glue import AwsGlueJobHook
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+import os.path
+
+
+class AWSGlueJobOperator(BaseOperator):
+"""
+Creates an AWS Glue Job. AWS Glue is a serverless Spark
+ETL service for running Spark Jobs on the AWS cloud.
+Language support: Python and Scala
+
+:param job_name: unique job name per AWS Account
+:type job_name: Optional[str]
+:param script_location: location of ETL script. Must be a local or S3 path
+:type script_location: Optional[str]
+:param job_desc: job description details
+:type job_desc: Optional[str]
+:param concurrent_run_limit: The maximum number of concurrent runs allowed
for a job
+:type concurrent_run_limit: Optional[int]
+:param script_args: etl script arguments and AWS Glue arguments
+:type script_args: dict
+:param connections: AWS Glue connections to be used by the job.
+:type connections: list
+:param retry_limit: The maximum number of times to retry this job if it
fails
+:type retry_limit:Optional[int]
+:param num_of_dpus: Number of AWS Glue DPUs to allocate to this Job.
+:type num_of_dpus: int
+:param region_name: aws region name (example: us-east-1)
+:type region_name: str
+:param s3_bucket: S3 bucket where logs and local etl script will be
uploaded
+:type s3_bucket: Optional[str]
+:param iam_role_name: AWS IAM Role for Glue Job Execution
+:type iam_role_name: Optional[str]
+"""
+template_fields = ()
+template_ext = ()
+ui_color = '#ededed'
+
+@apply_defaults
+def __init__(self,
+ job_name='aws_glue_default_job',
+ job_desc='AWS Glue Job with Airflow',
+ script_location=None,
+ concurrent_run_limit=None,
+ script_args=None,
+ connections=None,
+ retry_limit=None,
+ num_of_dpus=6,
+ aws_conn_id='aws_default',
+ region_name=None,
+ s3_bucket=None,
+ iam_role_name=None,
+ *args, **kwargs
+ ):
+super(AWSGlueJobOperator, self).__init__(*args, **kwargs)
+self.job_name = job_name
+self.job_desc = job_desc
+self.script_location = script_location
+self.concurrent_run_limit = concurrent_run_limit
+self.script_args = script_args or {}
+self.connections = connections or []
+self.retry_limit = retry_limit
+self.num_of_dpus = num_of_dpus
+self.aws_conn_id = aws_conn_id,
+self.region_name = region_name
+self.s3_bucket = s3_bucket
+self.iam_role_name = iam_role_name
+self.S3_PROTOCOL = "s3://"
+self.S3_ARTIFACTS_PREFIX = 'artifacts/glue-scripts/'
+
+def execute(self, context):
+"""
+Executes AWS Glue Job from Airflow
+
+:return: the id of the current glue job.
+"""
+if not self.script_location.startsWith(self.S3_PROTOCOL):
Review comment:
`self.script_location` should be an `arg` not a `kwarg`. It is required to
pass a script, isn't it? So we should not default it to `None`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407034741 ## File path: airflow/providers/google/cloud/hooks/dataflow.py ## @@ -585,10 +579,10 @@ def start_python_dataflow( # pylint: disable=too-many-arguments variables: Dict, dataflow: str, py_options: List[str], +project_id: str, Review comment: Ah.. Right :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407034827 ## File path: airflow/providers/google/cloud/hooks/dataflow.py ## @@ -585,10 +579,10 @@ def start_python_dataflow( # pylint: disable=too-many-arguments variables: Dict, dataflow: str, py_options: List[str], +project_id: str, Review comment: Totally forgot about it ! It was some 1.5 year ago! Thanks @mik-laj for reminding :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407034707 ## File path: airflow/providers/google/cloud/hooks/datacatalog.py ## @@ -659,7 +659,7 @@ def list_tags( location: str, entry_group: str, entry: str, -page_size: int = 100, +page_size: Optional[int] = 100, Review comment: OK. Making MyPy happy :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8145: Drop redundant project id null checks.
potiuk commented on a change in pull request #8145: Drop redundant project id null checks. URL: https://github.com/apache/airflow/pull/8145#discussion_r407034614 ## File path: airflow/providers/google/cloud/hooks/automl.py ## @@ -499,9 +481,6 @@ def update_dataset( :param update_mask: The update mask applies to the resource. If a dict is provided, it must be of the same form as the protobuf message FieldMask. :type update_mask: Union[dict, FieldMask] -:param project_id: ID of the Google Cloud project where dataset is located if None then Review comment: Right ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8249: DagCode is never updated
boring-cyborg[bot] commented on issue #8249: DagCode is never updated URL: https://github.com/apache/airflow/issues/8249#issuecomment-612363975 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] BasPH opened a new issue #8249: DagCode is never updated
BasPH opened a new issue #8249: DagCode is never updated URL: https://github.com/apache/airflow/issues/8249 **Apache Airflow version**: 1.10.10 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A **Environment**: docker-compose, LocalExecutor - **Cloud provider or hardware configuration**: - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: After making a change to my DAG, the values in the DagCode table are never updated, and the Code view always displays the code from the first time it was loaded. I do see my changes in the webserver in all other views. **What you expected to happen**: I expect to see the currently used code in the Code view at any point in time. **How to reproduce it**: **Anything else we need to know**: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] forbxy commented on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg
forbxy commented on issue #8237: dag will never start if we set a cron string with a clear day of week to the schedule_interval arg URL: https://github.com/apache/airflow/issues/8237#issuecomment-612357083 yes，dag never run at Friday or any day ，if i set a range of weekdays , like 1-5，it works. I think u can easily to recurrent it ，just set schedule_interval='12 14 * * 5' This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
