[jira] [Commented] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[ https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976213#comment-16976213 ] ASF subversion and git services commented on AIRFLOW-3489: -- Commit 942f6ad6b8701139f4d9a370949546f6fd328428 in airflow's branch refs/heads/master from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=942f6ad ] [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator (#6572) > PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly > - > > Key: AIRFLOW-3489 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3489 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Duan Shiqiang >Priority: Major > Labels: bigquery > > PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb) > as native python types (i.e. dictionary) to gcs new line separated json data. > But it generates bigquery schema for that field as data type string which > won't work if user want to import the gcs data into bigquery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[ https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976212#comment-16976212 ] ASF GitHub Bot commented on AIRFLOW-3489: - potiuk commented on pull request #6572: [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator URL: https://github.com/apache/airflow/pull/6572 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly > - > > Key: AIRFLOW-3489 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3489 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Duan Shiqiang >Priority: Major > Labels: bigquery > > PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb) > as native python types (i.e. dictionary) to gcs new line separated json data. > But it generates bigquery schema for that field as data type string which > won't work if user want to import the gcs data into bigquery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[ https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-3489. --- Fix Version/s: 2.0.0 Resolution: Fixed > PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly > - > > Key: AIRFLOW-3489 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3489 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Duan Shiqiang >Priority: Major > Labels: bigquery > Fix For: 2.0.0 > > > PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb) > as native python types (i.e. dictionary) to gcs new line separated json data. > But it generates bigquery schema for that field as data type string which > won't work if user want to import the gcs data into bigquery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6572: [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator
potiuk merged pull request #6572: [AIRFLOW-3489] Improve json data handling in PostgresToGcs operator URL: https://github.com/apache/airflow/pull/6572 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
URL: https://github.com/apache/airflow/pull/6595#issuecomment-554800995
And one more thing here @bolkedebruin - for some of those scripts (static
checks) we wanted to force always 3.5 version - no matter what you had setup as
your current python version (simply because we wanted to run static checks
consistently for all static checks. That's why in those scripts we "hard-coded"
3.5 before any of the common scripts were loaded, to make sure that
PYTHON_VERSION is set before any of the initialisation happens. I will have to
add a common script to force PYTHON_VERSION but I think it's a bit easier to
write:
```
PYTHON_VERSION=3.6
```
rather than
```
source ${ARIFLOW_SOURCES}/common/_force_python_version.sh
```
I am reviewing the scripts now in my PR and will try to find some good and
readable solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6002) Drop python 3.5 from support matrix
[ https://issues.apache.org/jira/browse/AIRFLOW-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976193#comment-16976193 ] ASF GitHub Bot commented on AIRFLOW-6002: - potiuk commented on pull request #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drop python 3.5 from support matrix > --- > > Key: AIRFLOW-6002 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6002 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.6 >Reporter: Bolke de Bruin >Priority: Major > > Python 3.5 is not the default anymore on Debian 's latest release and support > for it is holding back improved syntax and typing support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6002) Drop python 3.5 from support matrix
[ https://issues.apache.org/jira/browse/AIRFLOW-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976194#comment-16976194 ] ASF subversion and git services commented on AIRFLOW-6002: -- Commit 6a853db37b89a87cd829cb9dc9e3aea8ddcd671d in airflow's branch refs/heads/master from bolkedebruin [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6a853db ] [AIRFLOW-6002] Drop support for python 3.5 (#6595) > Drop python 3.5 from support matrix > --- > > Key: AIRFLOW-6002 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6002 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.6 >Reporter: Bolke de Bruin >Priority: Major > > Python 3.5 is not the default anymore on Debian 's latest release and support > for it is holding back improved syntax and typing support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6595: [AIRFLOW-6002] Drop support for python 3.5
potiuk merged pull request #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554798105 > > There is one more python 3.5 in breeze-complete > > Why were those harcoded everywhere @potiuk ? Sorry for that. That was unfortunate indeed. I have a refactor (WIP) for the whole set of bash scripts to make them way simpler (#6500) - currently we are at 8/12 PRs (https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-5606) done for the production image change that simplified the scripts (and I will make a DEFAULT_PYTHON_VERSION there in common utils script that we will use). Unfortunately this is not always super-easy for the bash scripts - especially that so far we had a separate build script for Dockerhub (hooks/build) - this is where the whole build logic was implemented. We called this script from within other scripts (and separately in Dockerhub), and it got far too complex. I reverted the logic in (#6500) so that we have common utils used everywhere and then it will be possible to make it defined in one place and have one constant for it (in bash scripts). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347165174 ## File path: airflow/executors/kubernetes_executor.py ## @@ -628,6 +629,7 @@ def clear_not_launched_queued_tasks(self, session=None): proper support for State.LAUNCHED """ +from airflow.models import TaskInstance Review comment: Import of model's Task Instance caused another cyclic import when we run full pylint serial check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347165003 ## File path: airflow/executors/dask_executor.py ## @@ -44,6 +42,7 @@ def __init__(self, cluster_address=None): super().__init__(parallelism=0) def start(self): +import distributed Review comment: After introduction suggestions from comment - https://github.com/apache/airflow/pull/6596/files#r347158027 , the problem will disappear. The dask_executor.py file will not be loaded if the executor is not selected, so the `distributed` library is probably not installed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347164833 ## File path: airflow/executors/__init__.py ## @@ -57,32 +61,21 @@ def get_default_executor(): return DEFAULT_EXECUTOR -class Executors: -LocalExecutor = "LocalExecutor" -SequentialExecutor = "SequentialExecutor" -CeleryExecutor = "CeleryExecutor" -DaskExecutor = "DaskExecutor" -KubernetesExecutor = "KubernetesExecutor" - - def _get_executor(executor_name): """ Creates a new instance of the named executor. In case the executor name is not know in airflow, look for it in the plugins """ -if executor_name == Executors.LocalExecutor: +if executor_name == AvailableExecutors.LOCAL_EXECUTOR: return LocalExecutor() -elif executor_name == Executors.SequentialExecutor: +elif executor_name == AvailableExecutors.SEQUENTIAL_EXECUTOR: return SequentialExecutor() -elif executor_name == Executors.CeleryExecutor: -from airflow.executors.celery_executor import CeleryExecutor +elif executor_name == AvailableExecutors.CELERY_EXECUTOR: Review comment: This has a big impact on performance because one class can load more classes, and each class must be initialized. Airflow separate tasks are run in separate processes, which means that classes are loaded repeatedly. If possible, we should limit the classes that can be loaded. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347164105 ## File path: airflow/executors/dask_executor.py ## @@ -44,6 +42,7 @@ def __init__(self, cluster_address=None): super().__init__(parallelism=0) def start(self): +import distributed Review comment: What do you mean? The 'distributed' package is added as dependency when 'dask' is installed. It is not installed even when "All" is configured. I'd love to get rid of dask executor - I think no-one uses it any more but this is a process that requires voting and community to agree. Right now I try to get rid of cyclic import problems and not get rid of dask executor. Do you have a suggestion what else can we do now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347163897 ## File path: airflow/executors/__init__.py ## @@ -57,32 +61,21 @@ def get_default_executor(): return DEFAULT_EXECUTOR -class Executors: -LocalExecutor = "LocalExecutor" -SequentialExecutor = "SequentialExecutor" -CeleryExecutor = "CeleryExecutor" -DaskExecutor = "DaskExecutor" -KubernetesExecutor = "KubernetesExecutor" - - def _get_executor(executor_name): """ Creates a new instance of the named executor. In case the executor name is not know in airflow, look for it in the plugins """ -if executor_name == Executors.LocalExecutor: +if executor_name == AvailableExecutors.LOCAL_EXECUTOR: return LocalExecutor() -elif executor_name == Executors.SequentialExecutor: +elif executor_name == AvailableExecutors.SEQUENTIAL_EXECUTOR: return SequentialExecutor() -elif executor_name == Executors.CeleryExecutor: -from airflow.executors.celery_executor import CeleryExecutor +elif executor_name == AvailableExecutors.CELERY_EXECUTOR: Review comment: Yes. It caused mutiple other problems. But it was wrong originally that only some of the executors were locally imported (and that was part of the problems with cyclic imports). Now all of them are locally imported. I don't think it has an impact on performance. The classes are too small to make a difference. I think optimising for performance here would be a premature optimisation. However cyclic imports should be avoided and I hope I can make all the tests succeed now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347163146 ## File path: airflow/executors/dask_executor.py ## @@ -44,6 +42,7 @@ def __init__(self, cluster_address=None): super().__init__(parallelism=0) def start(self): +import distributed Review comment: I think we should avoid the whole file if it's not needed. This causes the problems with importing non-existing libraries to disappear. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347162940 ## File path: airflow/executors/kubernetes_executor.py ## @@ -628,6 +629,7 @@ def clear_not_launched_queued_tasks(self, session=None): proper support for State.LAUNCHED """ +from airflow.models import TaskInstance Review comment: What is purpose of this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
codecov-io edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554792563 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=h1) Report > Merging [#6595](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5c2c0f003d469a6e131be906898899f9330298a8?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6595 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 647 647 Lines 37408 37408 = - Hits31359 31237 -122 - Misses 60496171 +122 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=footer). Last update [5c2c0f0...0b0cecf](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
codecov-io edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554792563 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=h1) Report > Merging [#6595](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5c2c0f003d469a6e131be906898899f9330298a8?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6595 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 647 647 Lines 37408 37408 = - Hits31359 31237 -122 - Misses 60496171 +122 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=footer). Last update [5c2c0f0...0b0cecf](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
codecov-io commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554792563 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=h1) Report > Merging [#6595](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5c2c0f003d469a6e131be906898899f9330298a8?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6595 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 647 647 Lines 37408 37408 = - Hits31359 31237 -122 - Misses 60496171 +122 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6595/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=footer). Last update [5c2c0f0...0b0cecf](https://codecov.io/gh/apache/airflow/pull/6595?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk opened a new pull request #6599: [AIRFLOW-6007] Check providers instead of provider package
potiuk opened a new pull request #6599: [AIRFLOW-6007] Check providers instead
of provider package
URL: https://github.com/apache/airflow/pull/6599
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6007
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6599: [AIRFLOW-6007] Check providers instead of provider package
potiuk commented on issue #6599: [AIRFLOW-6007] Check providers instead of provider package URL: https://github.com/apache/airflow/pull/6599#issuecomment-554791497 @mik-laj - seems we are not checking providers now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages
[
https://issues.apache.org/jira/browse/AIRFLOW-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976175#comment-16976175
]
ASF GitHub Bot commented on AIRFLOW-6007:
-
potiuk commented on pull request #6599: [AIRFLOW-6007] Check providers instead
of provider package
URL: https://github.com/apache/airflow/pull/6599
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6007
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Doc check checks provider/ instead of providers/ packages
> -
>
> Key: AIRFLOW-6007
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6007
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages
Jarek Potiuk created AIRFLOW-6007: - Summary: Doc check checks provider/ instead of providers/ packages Key: AIRFLOW-6007 URL: https://issues.apache.org/jira/browse/AIRFLOW-6007 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] bolkedebruin commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
bolkedebruin commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554786397 > There is one more python 3.5 in breeze-complete Why were those harcoded everywhere @potiuk ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6006) Move DAG out of airflow package imports
[
https://issues.apache.org/jira/browse/AIRFLOW-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976149#comment-16976149
]
ASF GitHub Bot commented on AIRFLOW-6006:
-
potiuk commented on pull request #6598: [AIRFLOW-6006] Remove dag out of
airflow package import. Depends on [AIRFLOW-6004] [AIRFLOW-6005]
URL: https://github.com/apache/airflow/pull/6598
Note to reviewer: Please take a look at the last commit.
moving DAG out of the airflow from all imports
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Move DAG out of airflow package imports
>
>
> Key: AIRFLOW-6006
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6006
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6598: [AIRFLOW-6006] Remove dag out of airflow package import. Depends on [AIRFLOW-6004] [AIRFLOW-6005]
potiuk opened a new pull request #6598: [AIRFLOW-6006] Remove dag out of
airflow package import. Depends on [AIRFLOW-6004] [AIRFLOW-6005]
URL: https://github.com/apache/airflow/pull/6598
Note to reviewer: Please take a look at the last commit.
moving DAG out of the airflow from all imports
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
mik-laj commented on a change in pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports URL: https://github.com/apache/airflow/pull/6596#discussion_r347158027 ## File path: airflow/executors/__init__.py ## @@ -57,32 +61,21 @@ def get_default_executor(): return DEFAULT_EXECUTOR -class Executors: -LocalExecutor = "LocalExecutor" -SequentialExecutor = "SequentialExecutor" -CeleryExecutor = "CeleryExecutor" -DaskExecutor = "DaskExecutor" -KubernetesExecutor = "KubernetesExecutor" - - def _get_executor(executor_name): """ Creates a new instance of the named executor. In case the executor name is not know in airflow, look for it in the plugins """ -if executor_name == Executors.LocalExecutor: +if executor_name == AvailableExecutors.LOCAL_EXECUTOR: return LocalExecutor() -elif executor_name == Executors.SequentialExecutor: +elif executor_name == AvailableExecutors.SEQUENTIAL_EXECUTOR: return SequentialExecutor() -elif executor_name == Executors.CeleryExecutor: -from airflow.executors.celery_executor import CeleryExecutor +elif executor_name == AvailableExecutors.CELERY_EXECUTOR: Review comment: It is very important that the modules are imported here. Otherwise, all possible executors will load on every person, which causes a drastic increase in the time it takes to run the application. This means that everyone must also have all executors installed. In this PR: https://github.com/apache/airflow/pull/6594 In this PR, I have reduced the number of modules loaded and you can see how it affected performance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554783431 There is one more python 3.5 in breeze-complete @bolkedebruin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #6588: [AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws
nuclearpinguin commented on issue #6588: [AIRFLOW-5781] AIP-21 Migrate AWS Kinesis to /providers/amazon/aws URL: https://github.com/apache/airflow/pull/6588#issuecomment-554783407 I think the CI should work after #6596 is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands
mik-laj edited a comment on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands URL: https://github.com/apache/airflow/pull/6594#issuecomment-554783165 @nuclearpinguin Sometimes there is one command and sometimes there are several commands. It depends if it is a single command or a subcommand. Subcommend is a command that lets you invoke other commands, but it is still a command, although it groups other commands, so I'd rather stay with the current prefix. This is a big change, but unfortunately it is very difficult to divide it into several PRs, because there will be a lot of conflicts in imports.I divided it very carefully into several commits so that everyone can do a review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands
mik-laj commented on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands URL: https://github.com/apache/airflow/pull/6594#issuecomment-554783165 @nuclearpinguin Sometimes there is one command and sometimes there are several commands. It depends if it is a single command or a subcommand. Subcommend is a command that lets you invoke other commands, but it is still a command, although it groups other commands, so I'd rather stay with the current prefix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bolkedebruin commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
bolkedebruin commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554781636 > We need a bit more than that. There are a number of bash scripts with PYTHON_VERSION set to 3.5. I will remove those. > I might do a commit with that on top of yours shortly. Please do not do that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bolkedebruin edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
bolkedebruin edited a comment on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554781636 > We need a bit more than that. There are a number of bash scripts with PYTHON_VERSION set to 3.5. I will remove those, I think that to be bad practice. > I might do a commit with that on top of yours shortly. Please do not do that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6006) Move DAG out of airflow package imports
Jarek Potiuk created AIRFLOW-6006: - Summary: Move DAG out of airflow package imports Key: AIRFLOW-6006 URL: https://issues.apache.org/jira/browse/AIRFLOW-6006 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
m1racoli commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#issuecomment-554776399 Hey @OmerJog, I just came back from vacation. Yes, I'll work on this PR. I should find some time in the next weeks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6005) Remove all imports of AirflowException directly from airflow package
[
https://issues.apache.org/jira/browse/AIRFLOW-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976129#comment-16976129
]
ASF GitHub Bot commented on AIRFLOW-6005:
-
potiuk commented on pull request #6597: [AIRFLOW-6005] Remove airflow airflow
exception imports. Depends on [AIRFLOW-6004]
URL: https://github.com/apache/airflow/pull/6597
NOTE FOR REVIEWER: This change depends on #6596 so please review only the
last commit.
As explained in https://issues.apache.org/jira/browse/AIRFLOW-6003
having classes/packages in "airflow" might lead to cyclic imports
and is generally a bad practice. It makes more sense to import
those packages from where they belong - i.e. sub-packages of airflow.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove all imports of AirflowException directly from airflow package
>
>
> Key: AIRFLOW-6005
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6005
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6597: [AIRFLOW-6005] Remove airflow airflow exception imports. Depends on [AIRFLOW-6004]
potiuk opened a new pull request #6597: [AIRFLOW-6005] Remove airflow airflow
exception imports. Depends on [AIRFLOW-6004]
URL: https://github.com/apache/airflow/pull/6597
NOTE FOR REVIEWER: This change depends on #6596 so please review only the
last commit.
As explained in https://issues.apache.org/jira/browse/AIRFLOW-6003
having classes/packages in "airflow" might lead to cyclic imports
and is generally a bad practice. It makes more sense to import
those packages from where they belong - i.e. sub-packages of airflow.
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6005) Remove all imports of AirflowException directly from airflow package
Jarek Potiuk created AIRFLOW-6005: - Summary: Remove all imports of AirflowException directly from airflow package Key: AIRFLOW-6005 URL: https://issues.apache.org/jira/browse/AIRFLOW-6005 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default
codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default URL: https://github.com/apache/airflow/pull/4465#issuecomment-452625380 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=h1) Report > Merging [#4465](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/109789283dfda76649fafb7535fee82300e95d28?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4465 +/- ## == - Coverage 83.82% 83.51% -0.32% == Files 647 647 Lines 3740837408 == - Hits3135931241 -118 - Misses 6049 6167 +118 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) | Coverage Δ | | |---|---|---| | [.../contrib/operators/emr\_create\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfY3JlYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `92% <ø> (ø)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `95.65% <ø> (ø)` | :arrow_up: | | [...ntrib/operators/emr\_terminate\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfdGVybWluYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `95% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.13% <0%> (-0.31%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=footer). Last update [1097892...a034d85](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default
codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default URL: https://github.com/apache/airflow/pull/4465#issuecomment-452625380 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=h1) Report > Merging [#4465](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/109789283dfda76649fafb7535fee82300e95d28?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4465 +/- ## == - Coverage 83.82% 83.51% -0.32% == Files 647 647 Lines 3740837408 == - Hits3135931241 -118 - Misses 6049 6167 +118 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) | Coverage Δ | | |---|---|---| | [.../contrib/operators/emr\_create\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfY3JlYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `92% <ø> (ø)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `95.65% <ø> (ø)` | :arrow_up: | | [...ntrib/operators/emr\_terminate\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfdGVybWluYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `95% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `77.14% <0%> (-21.43%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.13% <0%> (-0.31%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=footer). Last update [1097892...a034d85](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
[
https://issues.apache.org/jira/browse/AIRFLOW-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976123#comment-16976123
]
ASF GitHub Bot commented on AIRFLOW-6004:
-
potiuk commented on pull request #6596: [AIRFLOW-6004] Untangle Executors class
to avoid cyclic imports
URL: https://github.com/apache/airflow/pull/6596
There are cyclic imports detected seemingly randomly by pylint checks when
some
of the PRs are run in CI:
* Module airflow.utils.log.json_formatter
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
The problem is that airflow's _init_ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor._init_.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor)
import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number
of
files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module.
Also the name of constants was changed to not to be confused with class
names and Executors class was renamed to AvailableExecutors.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6004
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Untangle "Executors" class from potentially cyclic import
> -
>
> Key: AIRFLOW-6004
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6004
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> See the description in https://issues.apache.org/jira/browse/AIRFLOW-6003
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports
potiuk opened a new pull request #6596: [AIRFLOW-6004] Untangle Executors class
to avoid cyclic imports
URL: https://github.com/apache/airflow/pull/6596
There are cyclic imports detected seemingly randomly by pylint checks when
some
of the PRs are run in CI:
* Module airflow.utils.log.json_formatter
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
The problem is that airflow's _init_ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor._init_.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor)
import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number
of
files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module.
Also the name of constants was changed to not to be confused with class
names and Executors class was renamed to AvailableExecutors.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6004
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
[ https://issues.apache.org/jira/browse/AIRFLOW-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6004: -- Description: See the description in https://issues.apache.org/jira/browse/AIRFLOW-6003 (was: See the description in ) > Untangle "Executors" class from potentially cyclic import > - > > Key: AIRFLOW-6004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > See the description in https://issues.apache.org/jira/browse/AIRFLOW-6003 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
Jarek Potiuk created AIRFLOW-6004: - Summary: Untangle "Executors" class from potentially cyclic import Key: AIRFLOW-6004 URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
[ https://issues.apache.org/jira/browse/AIRFLOW-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6004: -- Description: See the description in > Untangle "Executors" class from potentially cyclic import > - > > Key: AIRFLOW-6004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > See the description in -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6003) Cyclic imports detected by pylint randomly
[
https://issues.apache.org/jira/browse/AIRFLOW-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk updated AIRFLOW-6003:
--
Description:
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI:
An example of such pylint problem:
[https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status]
{code:java}
* Module airflow.utils.log.json_formatter
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
{code}
The problem is that airflow's __init__ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor.__init__.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor) import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number
of files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module and a
number of classes (such as AirflowException) should be moved out from __init__
package and imported from where they belong (so for example AirflowException
should be universally imported from airflow.exceptions).
was:
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI:
An example of such pylint problem:
[https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status]
* Module airflow.utils.log.json_formatter
1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
The problem is that airflow's __init__ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor.__init__.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor) import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number of
files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module and a
number of classes (such as AirflowException) should be moved out from __init__
package and imported from where they belong (so for example AirflowException
should be universally imported from airflow.exceptions).
> Cyclic imports detected by pylint randomly
> --
>
> Key: AIRFLOW-6003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6003
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
>Affects Versions: 2.0.0
>Reporter: Jarek Potiuk
>Priority: Major
>
> There are cyclic imports detected seemingly randomly by pylint checks when
> some of the PRs are run in CI:
> An example of such pylint problem:
> [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status]
>
>
> {code:java}
> * Module airflow.utils.log.json_formatter
>
[jira] [Created] (AIRFLOW-6003) Cyclic imports detected by pylint randomly
Jarek Potiuk created AIRFLOW-6003: - Summary: Cyclic imports detected by pylint randomly Key: AIRFLOW-6003 URL: https://issues.apache.org/jira/browse/AIRFLOW-6003 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI: An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status] * Module airflow.utils.log.json_formatter 1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import) 1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import) 1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import) The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order. As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands
codecov-io commented on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands URL: https://github.com/apache/airflow/pull/6594#issuecomment-554772240 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=h1) Report > Merging [#6594](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5c2c0f003d469a6e131be906898899f9330298a8?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `65.98%`. [](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6594 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 647 665 +18 Lines 37408 37537 +129 = - Hits31359 31346 -13 - Misses 60496191 +142 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/cli/commands/rotate\_fernet\_key\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcm90YXRlX2Zlcm5ldF9rZXlfY29tbWFuZC5weQ==) | `0% <0%> (ø)` | | | [airflow/cli/commands/serve\_logs\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2VydmVfbG9nc19jb21tYW5kLnB5) | `0% <0%> (ø)` | | | [airflow/cli/commands/scheduler\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2NoZWR1bGVyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | | | [airflow/cli/commands/kerberos\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMva2VyYmVyb3NfY29tbWFuZC5weQ==) | `0% <0%> (ø)` | | | [airflow/cli/commands/flower\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZmxvd2VyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | | | [airflow/api/client/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L19faW5pdF9fLnB5) | `100% <100%> (ø)` | | | [airflow/cli/commands/role\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcm9sZV9jb21tYW5kLnB5) | `100% <100%> (ø)` | | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `96% <100%> (+0.16%)` | :arrow_up: | | [airflow/cli/commands/version\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdmVyc2lvbl9jb21tYW5kLnB5) | `100% <100%> (ø)` | | | [airflow/cli/commands/sync\_perm\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc3luY19wZXJtX2NvbW1hbmQucHk=) | `100% <100%> (ø)` | | | ... and [37 more](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=footer). Last update [5c2c0f0...911910e](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands
codecov-io edited a comment on issue #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands URL: https://github.com/apache/airflow/pull/6594#issuecomment-554772240 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=h1) Report > Merging [#6594](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/5c2c0f003d469a6e131be906898899f9330298a8?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `65.98%`. [](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6594 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 647 665 +18 Lines 37408 37537 +129 = - Hits31359 31346 -13 - Misses 60496191 +142 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/cli/commands/rotate\_fernet\_key\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcm90YXRlX2Zlcm5ldF9rZXlfY29tbWFuZC5weQ==) | `0% <0%> (ø)` | | | [airflow/cli/commands/serve\_logs\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2VydmVfbG9nc19jb21tYW5kLnB5) | `0% <0%> (ø)` | | | [airflow/cli/commands/scheduler\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc2NoZWR1bGVyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | | | [airflow/cli/commands/kerberos\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMva2VyYmVyb3NfY29tbWFuZC5weQ==) | `0% <0%> (ø)` | | | [airflow/cli/commands/flower\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZmxvd2VyX2NvbW1hbmQucHk=) | `0% <0%> (ø)` | | | [airflow/api/client/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY2xpZW50L19faW5pdF9fLnB5) | `100% <100%> (ø)` | | | [airflow/cli/commands/role\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcm9sZV9jb21tYW5kLnB5) | `100% <100%> (ø)` | | | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `96% <100%> (+0.16%)` | :arrow_up: | | [airflow/cli/commands/version\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdmVyc2lvbl9jb21tYW5kLnB5) | `100% <100%> (ø)` | | | [airflow/cli/commands/sync\_perm\_command.py](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvc3luY19wZXJtX2NvbW1hbmQucHk=) | `100% <100%> (ø)` | | | ... and [37 more](https://codecov.io/gh/apache/airflow/pull/6594/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=footer). Last update [5c2c0f0...911910e](https://codecov.io/gh/apache/airflow/pull/6594?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5
potiuk commented on issue #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595#issuecomment-554770977 We need a bit more than that. There are a number of bash scripts with PYTHON_VERSION set to 3.5. I might do a commit with that on top of yours shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6002) Drop python 3.5 from support matrix
[ https://issues.apache.org/jira/browse/AIRFLOW-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976103#comment-16976103 ] ASF GitHub Bot commented on AIRFLOW-6002: - bolkedebruin commented on pull request #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595 Make sure you have checked _all_ steps below. ### Jira - [X] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-6002 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: Drop python 3.5 support per mailinglist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drop python 3.5 from support matrix > --- > > Key: AIRFLOW-6002 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6002 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.6 >Reporter: Bolke de Bruin >Priority: Major > > Python 3.5 is not the default anymore on Debian 's latest release and support > for it is holding back improved syntax and typing support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] bolkedebruin opened a new pull request #6595: [AIRFLOW-6002] Drop support for python 3.5
bolkedebruin opened a new pull request #6595: [AIRFLOW-6002] Drop support for python 3.5 URL: https://github.com/apache/airflow/pull/6595 Make sure you have checked _all_ steps below. ### Jira - [X] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-6002 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. - In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)). - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Description - [X] Here are some details about my PR, including screenshots of any UI changes: Drop python 3.5 support per mailinglist. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6002) Drop python 3.5 from support matrix
Bolke de Bruin created AIRFLOW-6002: --- Summary: Drop python 3.5 from support matrix Key: AIRFLOW-6002 URL: https://issues.apache.org/jira/browse/AIRFLOW-6002 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.6 Reporter: Bolke de Bruin Python 3.5 is not the default anymore on Debian 's latest release and support for it is holding back improved syntax and typing support -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #6526: AIRFLOW-2143a - Fix TaskTries graph counts off-by-1
codecov-io commented on issue #6526: AIRFLOW-2143a - Fix TaskTries graph counts off-by-1 URL: https://github.com/apache/airflow/pull/6526#issuecomment-554767592 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=h1) Report > Merging [#6526](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/713323d818eaa3a96ab0cf47703651e0cc1a29b0?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `57.14%`. [](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#6526 +/- ## == + Coverage 79.63% 79.63% +<.01% == Files 518 518 Lines 3526935267 -2 == Hits2808628086 + Misses 7183 7181 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6526/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `45.03% <0%> (+0.05%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6526/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.67% <100%> (+0.02%)` | :arrow_up: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/airflow/pull/6526/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `75.61% <50%> (+0.1%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6526/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85.71% <0%> (-4.77%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6526/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `59.02% <0%> (+0.32%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=footer). Last update [713323d...66394c0](https://codecov.io/gh/apache/airflow/pull/6526?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default
codecov-io edited a comment on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default URL: https://github.com/apache/airflow/pull/4465#issuecomment-452625380 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=h1) Report > Merging [#4465](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/109789283dfda76649fafb7535fee82300e95d28?src=pr=desc) will **decrease** coverage by `9.56%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4465 +/- ## == - Coverage 83.82% 74.26% -9.57% == Files 647 430 -217 Lines 3740827966-9442 == - Hits3135920768 -10591 - Misses 6049 7198+1149 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=tree) | Coverage Δ | | |---|---|---| | [.../contrib/operators/emr\_create\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfY3JlYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `92% <ø> (ø)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `95.65% <ø> (ø)` | :arrow_up: | | [...ntrib/operators/emr\_terminate\_job\_flow\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfdGVybWluYXRlX2pvYl9mbG93X29wZXJhdG9yLnB5) | `95% <ø> (ø)` | :arrow_up: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...rflow/contrib/operators/bigquery\_check\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9jaGVja19vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/bigquery\_get\_data.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9iaWdxdWVyeV9nZXRfZGF0YS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...low/contrib/operators/datastore\_import\_operator.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhc3RvcmVfaW1wb3J0X29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...irflow/contrib/example\_dags/example\_twitter\_dag.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3R3aXR0ZXJfZGFnLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...trib/example\_dags/example\_azure\_cosmosdb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2F6dXJlX2Nvc21vc2RiX3NlbnNvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | ... and [603 more](https://codecov.io/gh/apache/airflow/pull/4465/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=footer). Last update [1097892...a034d85](https://codecov.io/gh/apache/airflow/pull/4465?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] laakejs commented on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default
laakejs commented on issue #4465: [AIRFLOW-3682] Rename s3_default to aws_default URL: https://github.com/apache/airflow/pull/4465#issuecomment-554766689 @OmerJog I had other work to do and, to be honest, forgot this open pull request. Thank you for the friendly reminder. I fixed the issues @Fokko mentioned above. From my side there is nothing else left to do, or am I mistaken? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3754) Google Cloud Storage Hook to take in Stream/String from an URL
[ https://issues.apache.org/jira/browse/AIRFLOW-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-3754: --- Component/s: gcp > Google Cloud Storage Hook to take in Stream/String from an URL > -- > > Key: AIRFLOW-3754 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3754 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks >Reporter: David Choy >Priority: Minor > > Hi all, > It would be nice to have a Google Cloud Storage Hook to take in Stream/String > from an URL. > Thanks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6001) Lazy load CLI commands
[
https://issues.apache.org/jira/browse/AIRFLOW-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976083#comment-16976083
]
ASF GitHub Bot commented on AIRFLOW-6001:
-
mik-laj commented on pull request #6594: [AIRFLOW-6001][depends on
AIRFLOW-6000] Lazy load CLI commands
URL: https://github.com/apache/airflow/pull/6594
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6001
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lazy load CLI commands
> --
>
> Key: AIRFLOW-6001
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6001
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>
> Currently, CLI loads all packages even when the command is not used in any
> way, which takes CPU time and memory. I suggest that only commands that are
> executed be loaded. This can speed up task launch time and also improve user
> productivity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #5297: AIRFLOW-2143 - Fix TaskTries graph counts off-by-1
codecov-io edited a comment on issue #5297: AIRFLOW-2143 - Fix TaskTries graph counts off-by-1 URL: https://github.com/apache/airflow/pull/5297#issuecomment-493643996 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=h1) Report > Merging [#5297](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/109789283dfda76649fafb7535fee82300e95d28?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `80%`. [](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5297 +/- ## == - Coverage 83.82% 83.82% -0.01% == Files 647 647 Lines 3740837408 == - Hits3135931356 -3 - Misses 6049 6052 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5297/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.81% <100%> (+0.02%)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5297/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.58% <50%> (+0.1%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/5297/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `89.9% <0%> (-1.53%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=footer). Last update [1097892...1ff771c](https://codecov.io/gh/apache/airflow/pull/5297?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj opened a new pull request #6594: [AIRFLOW-6001][depends on AIRFLOW-6000] Lazy load CLI commands
mik-laj opened a new pull request #6594: [AIRFLOW-6001][depends on
AIRFLOW-6000] Lazy load CLI commands
URL: https://github.com/apache/airflow/pull/6594
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6001
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6001) Lazy load CLI commands
Kamil Bregula created AIRFLOW-6001: -- Summary: Lazy load CLI commands Key: AIRFLOW-6001 URL: https://issues.apache.org/jira/browse/AIRFLOW-6001 Project: Apache Airflow Issue Type: Bug Components: cli Affects Versions: 1.10.6 Reporter: Kamil Bregula Currently, CLI loads all packages even when the command is not used in any way, which takes CPU time and memory. I suggest that only commands that are executed be loaded. This can speed up task launch time and also improve user productivity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5923) Use absolute paths in GCP system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976081#comment-16976081 ] ASF subversion and git services commented on AIRFLOW-5923: -- Commit 5c2c0f003d469a6e131be906898899f9330298a8 in airflow's branch refs/heads/master from Tomek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5c2c0f0 ] [AIRFLOW-5923] Use absolute paths in GCP system tests (#6571) > Use absolute paths in GCP system tests > -- > > Key: AIRFLOW-5923 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5923 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5923) Use absolute paths in GCP system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5923. --- Fix Version/s: 2.0.0 Resolution: Fixed > Use absolute paths in GCP system tests > -- > > Key: AIRFLOW-5923 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5923 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5923) Use absolute paths in GCP system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976080#comment-16976080 ] ASF GitHub Bot commented on AIRFLOW-5923: - potiuk commented on pull request #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests URL: https://github.com/apache/airflow/pull/6571 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Use absolute paths in GCP system tests > -- > > Key: AIRFLOW-5923 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5923 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests
potiuk merged pull request #6571: [AIRFLOW-5923] Use absolute paths in GCP system tests URL: https://github.com/apache/airflow/pull/6571 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r347145094 ## File path: airflow/ti_deps/deps/trigger_rule_dep.py ## @@ -49,33 +75,33 @@ def _get_dep_statuses(self, ti, session, dep_context): yield self._passing_status(reason="The task had a dummy trigger rule set.") return -# TODO(unknown): this query becomes quite expensive with dags that have many -# tasks. It should be refactored to let the task report to the dag run and get the -# aggregates from there. -qry = ( -session -.query( -func.coalesce(func.sum( -case([(TI.state == State.SUCCESS, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.SKIPPED, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.FAILED, 1)], else_=0)), 0), -func.coalesce(func.sum( -case([(TI.state == State.UPSTREAM_FAILED, 1)], else_=0)), 0), -func.count(TI.task_id), +if dep_context.finished_tasks is None: Review comment: I will check this, I agree This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r347145103 ## File path: airflow/ti_deps/deps/trigger_rule_dep.py ## @@ -34,6 +35,31 @@ class TriggerRuleDep(BaseTIDep): IGNOREABLE = True IS_TASK_DEP = True +@staticmethod +def _get_states_count_upstream_ti(ti, finished_tasks): +""" +:param ti the ti that we want to calculate deps for +:type ti airflow.models.TaskInstance +:param finished_tasks all the finished tasks of the dag_run +:type finished_tasks of finished ti's +""" +successes, skipped, failed, upstream_failed, done = 0, 0, 0, 0, 0 +upstream_tasks = [finished_task for finished_task in finished_tasks + if finished_task.task_id in ti.task.upstream_task_ids] +if upstream_tasks: +upstream_tasks_sorted = sorted(upstream_tasks, key=lambda x: x.state) +for k, g in groupby(upstream_tasks_sorted, key=lambda x: x.state): +if k == State.SUCCESS: +successes = len(list(g)) +elif k == State.SKIPPED: +skipped = len(list(g)) +elif k == State.FAILED: +failed = len(list(g)) +elif k == State.UPSTREAM_FAILED: +upstream_failed = len(list(g)) Review comment: Thanks, sounds neat I'l give it a shot. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-5941) MySQLHook initialization fails when db charset is utf8mb4
[
https://issues.apache.org/jira/browse/AIRFLOW-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976054#comment-16976054
]
Tomasz Żukowski edited comment on AIRFLOW-5941 at 11/17/19 2:59 PM:
I think it is unrelated as MySQLHook uses MySQLdb connection, not sqlqlchemy,
but fixing AIRFLOW-4824 will most probably solve my issue. Thanks for pointing.
was (Author: zuku):
I think it is unrelated as MySQLHook uses MySQLdb connection, not sqlqlchemy,
but fixing AIRFLOW-4824 will most probably solve my issue.
> MySQLHook initialization fails when db charset is utf8mb4
> -
>
> Key: AIRFLOW-5941
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5941
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.6
> Environment: AWS EC2 instance
>Reporter: Tomasz Żukowski
>Priority: Minor
>
> When trying to get the connection from MySQLHook below error is raised.
> Airflow version - 1.10.6
> MySQL version - 8.0.15
> MySQLdb (mysqlclient) - tested with 1.3.14(both installed with pip and built
> locally) and 1.4.2.post1
> connection extra:
> {code:python}
> {"charset":"utf8mb4"}
> {code}
> DB charset is set to utf8mb4
> Error message:
> {code:python}
> [2019-11-15 16:55:46,477] {taskinstance.py:1058} ERROR - (2006, "Can't
> initialize character set unknown (path: compiled_in)")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
> 930, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 113, in execute
> return_value = self.execute_callable()
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 118, in execute_callable
> return self.python_callable(*self.op_args, **self.op_kwargs)
> File "", line 136, in mysql_***
> mysql_conn = mysql_hook.get_conn()
> File "/usr/local/lib/python3.7/site-packages/airflow/hooks/mysql_hook.py",
> line 116, in get_conn
> conn = MySQLdb.connect(**conn_config)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/__init__.py",
> line 85, in Connect
> return Connection(*args, **kwargs)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/connections.py",
> line 208, in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> _mysql_exceptions.OperationalError: (2006, "Can't initialize character set
> unknown (path: compiled_in)")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5941) MySQLHook initialization fails when db charset is utf8mb4
[
https://issues.apache.org/jira/browse/AIRFLOW-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976054#comment-16976054
]
Tomasz Żukowski commented on AIRFLOW-5941:
--
I think it is unrelated as MySQLHook uses MySQLdb connection, not sqlqlchemy,
but fixing AIRFLOW-4824 will most probably solve my issue.
> MySQLHook initialization fails when db charset is utf8mb4
> -
>
> Key: AIRFLOW-5941
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5941
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.6
> Environment: AWS EC2 instance
>Reporter: Tomasz Żukowski
>Priority: Minor
>
> When trying to get the connection from MySQLHook below error is raised.
> Airflow version - 1.10.6
> MySQL version - 8.0.15
> MySQLdb (mysqlclient) - tested with 1.3.14(both installed with pip and built
> locally) and 1.4.2.post1
> connection extra:
> {code:python}
> {"charset":"utf8mb4"}
> {code}
> DB charset is set to utf8mb4
> Error message:
> {code:python}
> [2019-11-15 16:55:46,477] {taskinstance.py:1058} ERROR - (2006, "Can't
> initialize character set unknown (path: compiled_in)")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
> 930, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 113, in execute
> return_value = self.execute_callable()
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 118, in execute_callable
> return self.python_callable(*self.op_args, **self.op_kwargs)
> File "", line 136, in mysql_***
> mysql_conn = mysql_hook.get_conn()
> File "/usr/local/lib/python3.7/site-packages/airflow/hooks/mysql_hook.py",
> line 116, in get_conn
> conn = MySQLdb.connect(**conn_config)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/__init__.py",
> line 85, in Connect
> return Connection(*args, **kwargs)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/connections.py",
> line 208, in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> _mysql_exceptions.OperationalError: (2006, "Can't initialize character set
> unknown (path: compiled_in)")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-4824) MySqlHook needs to override DbApiHook.get_uri to pull in extra for charset=utf-8 during create_engine
[ https://issues.apache.org/jira/browse/AIRFLOW-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Żukowski reassigned AIRFLOW-4824: Assignee: Tomasz Żukowski > MySqlHook needs to override DbApiHook.get_uri to pull in extra for > charset=utf-8 during create_engine > - > > Key: AIRFLOW-4824 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4824 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.3 >Reporter: lolcode >Assignee: Tomasz Żukowski >Priority: Minor > > When using the engine from a MySQLHook in other code (such as Pandas) the > engine returned from the create_engine function is missing the charset=utf-8 > This issue was reported here: > [https://stackoverflow.com/questions/46084744/how-to-explicitly-declare-charset-utf8-for-airflow-connections] > conn = MySqlHook(mysql_conn_id='conn_id') > engine = conn.get_sqlalchemy_engine() > I can see that the code in function *get_uri* in dbapi_hook.py does not use > the charset = utf8 information from the extra section and that mysql_hook.py > does not override the function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3344) Airflow DAG object clear function does not clear tasks in the upstream_failed state when only_failed=True
[
https://issues.apache.org/jira/browse/AIRFLOW-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976043#comment-16976043
]
Steve Jacobs commented on AIRFLOW-3344:
---
I'm travelling this week so that means I'll either have plenty of time to PR a
change up or 0 time.
> Airflow DAG object clear function does not clear tasks in the upstream_failed
> state when only_failed=True
> -
>
> Key: AIRFLOW-3344
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3344
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.8.2, 1.9.0, 1.10.0
>Reporter: Steve Jacobs
>Priority: Minor
> Labels: easyfix, newbie
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When using the airflow clear command from the cli, you can pass an argument
> --only_failed to clear only failed tasks. This will clear ONLY tasks with the
> state failed, and not tasks with the state upstream_failed, causing any clear
> to still fail the dag_run if any upstream tasks are failed.
> Since one_failed as a trigger rule also checks for upstream_failed tasks, it
> seems consistent that this should also clear upstream_failed tasks. The
> relevant code change necessary is here:
> {code:java}
> if only_failed:
> tis = tis.filter(TI.state == State.FAILED)
> {code}
> to
> {code:java}
> if only_failed:
> tis = tis.filter(TI.state.in_([State.FAILED, State.UPSTREAM_FAILED]))
> {code}
> in models.py
> Additionally when clearing dags, the dag_run is set to the running state, but
> the dag_run start_date is not updated to the current time, as it is when
> clearing tasks through the Web UI. This causes dag_runs to fail on their
> timeouts even if the dag is full of successful tasks. This needs to be
> changed as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5940) Remove redundant Code in models.dagrun
[
https://issues.apache.org/jira/browse/AIRFLOW-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk resolved AIRFLOW-5940.
---
Fix Version/s: 2.0.0
Resolution: Resolved
> Remove redundant Code in models.dagrun
> --
>
> Key: AIRFLOW-5940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5940
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The below code in models.dagrun contains redundant code
> {code:python}
> is_task_in_dag = task is not None
> should_restore_task = is_task_in_dag and ti.state == State.REMOVED
> {code}
> It can be simplified by:
> {code:python}
> should_restore_task = (task is not None) and ti.state == State.REMOVED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5940) Remove redundant Code in models.dagrun
[
https://issues.apache.org/jira/browse/AIRFLOW-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976026#comment-16976026
]
ASF subversion and git services commented on AIRFLOW-5940:
--
Commit 109789283dfda76649fafb7535fee82300e95d28 in airflow's branch
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1097892 ]
[AIRFLOW-5940] Remove redundant Code in models.dagrun (#6591)
> Remove redundant Code in models.dagrun
> --
>
> Key: AIRFLOW-5940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5940
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 2.0.0
>
>
> The below code in models.dagrun contains redundant code
> {code:python}
> is_task_in_dag = task is not None
> should_restore_task = is_task_in_dag and ti.state == State.REMOVED
> {code}
> It can be simplified by:
> {code:python}
> should_restore_task = (task is not None) and ti.state == State.REMOVED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6591: [AIRFLOW-5940] Remove redundant Code in models.dagrun
potiuk merged pull request #6591: [AIRFLOW-5940] Remove redundant Code in models.dagrun URL: https://github.com/apache/airflow/pull/6591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5940) Remove redundant Code in models.dagrun
[
https://issues.apache.org/jira/browse/AIRFLOW-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976025#comment-16976025
]
ASF GitHub Bot commented on AIRFLOW-5940:
-
potiuk commented on pull request #6591: [AIRFLOW-5940] Remove redundant Code in
models.dagrun
URL: https://github.com/apache/airflow/pull/6591
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove redundant Code in models.dagrun
> --
>
> Key: AIRFLOW-5940
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5940
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Trivial
>
> The below code in models.dagrun contains redundant code
> {code:python}
> is_task_in_dag = task is not None
> should_restore_task = is_task_in_dag and ti.state == State.REMOVED
> {code}
> It can be simplified by:
> {code:python}
> should_restore_task = (task is not None) and ti.state == State.REMOVED
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976003#comment-16976003 ] jack commented on AIRFLOW-5920: --- [~tfindlay] you can open a Draft PR and get comments from contributors. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5232) Create pagerdutyhook in the airflow plugins, to support pagerduty.
[ https://issues.apache.org/jira/browse/AIRFLOW-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975997#comment-16975997 ] jack commented on AIRFLOW-5232: --- pagerdury hook was added in https://issues.apache.org/jira/browse/AIRFLOW-5832 This Jira can be closed > Create pagerdutyhook in the airflow plugins, to support pagerduty. > -- > > Key: AIRFLOW-5232 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5232 > Project: Apache Airflow > Issue Type: Wish > Components: plugins >Affects Versions: 1.10.5 >Reporter: sri ram chegondi >Priority: Major > > Create pagerdutyhook in the airflow plugins, to support pagerduty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3344) Airflow DAG object clear function does not clear tasks in the upstream_failed state when only_failed=True
[
https://issues.apache.org/jira/browse/AIRFLOW-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975993#comment-16975993
]
jack commented on AIRFLOW-3344:
---
[~steveatbat] can you PR your suggested change?
> Airflow DAG object clear function does not clear tasks in the upstream_failed
> state when only_failed=True
> -
>
> Key: AIRFLOW-3344
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3344
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.8.2, 1.9.0, 1.10.0
>Reporter: Steve Jacobs
>Priority: Minor
> Labels: easyfix, newbie
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> When using the airflow clear command from the cli, you can pass an argument
> --only_failed to clear only failed tasks. This will clear ONLY tasks with the
> state failed, and not tasks with the state upstream_failed, causing any clear
> to still fail the dag_run if any upstream tasks are failed.
> Since one_failed as a trigger rule also checks for upstream_failed tasks, it
> seems consistent that this should also clear upstream_failed tasks. The
> relevant code change necessary is here:
> {code:java}
> if only_failed:
> tis = tis.filter(TI.state == State.FAILED)
> {code}
> to
> {code:java}
> if only_failed:
> tis = tis.filter(TI.state.in_([State.FAILED, State.UPSTREAM_FAILED]))
> {code}
> in models.py
> Additionally when clearing dags, the dag_run is set to the running state, but
> the dag_run start_date is not updated to the current time, as it is when
> clearing tasks through the Web UI. This causes dag_runs to fail on their
> timeouts even if the dag is full of successful tasks. This needs to be
> changed as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-918) Improve bulk_load function for MySqlHook
[ https://issues.apache.org/jira/browse/AIRFLOW-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975988#comment-16975988 ] jack commented on AIRFLOW-918: -- Duplicate of https://issues.apache.org/jira/browse/AIRFLOW-5921 [~feluelle] no need for two Jiras > Improve bulk_load function for MySqlHook > > > Key: AIRFLOW-918 > URL: https://issues.apache.org/jira/browse/AIRFLOW-918 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.7.1.3 >Reporter: Ali Uz >Priority: Minor > Labels: easyfix, patch > > I think we can improve the `bulk_load` function in MySqlHook by adding a few > more parameters. For example, if I want to run a LOAD DATA command like the > following: > ``` > LOAD DATA LOCAL INFILE 'abc.csv' INTO TABLE abc > FIELDS TERMINATED BY ',' > ENCLOSED BY '"' > LINES TERMINATED BY '\r\n' > IGNORE 1 LINES > ``` > I would expect to supply the delimiter parameters, enclosing quotes > parameter, line terminating parameter and ignore line number parameter. > The current function only applies the following command: > ``` > LOAD DATA LOCAL INFILE 'abc.csv' INTO TABLE abc > ``` > It would be great if we could extend it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5616) PrestoHook to use prestodb
[ https://issues.apache.org/jira/browse/AIRFLOW-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975985#comment-16975985 ] jack commented on AIRFLOW-5616: --- By the way PrestoSQL might be better than Prestodb [https://github.com/prestosql/presto/issues/380] > PrestoHook to use prestodb > -- > > Key: AIRFLOW-5616 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5616 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.5 >Reporter: Alexandre Brilhante >Priority: Minor > > PrestoHook currently uses PyHive which doesn't support transactions whereas > prestodb > ([https://github.com/prestodb/presto-python-client)|https://github.com/prestodb/presto-python-client] > does. I think it would more flexible to use prestodb as client. I can work > on a PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5941) MySQLHook initialization fails when db charset is utf8mb4
[
https://issues.apache.org/jira/browse/AIRFLOW-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975982#comment-16975982
]
jack commented on AIRFLOW-5941:
---
Possible root cause is https://issues.apache.org/jira/browse/AIRFLOW-4824 ?
> MySQLHook initialization fails when db charset is utf8mb4
> -
>
> Key: AIRFLOW-5941
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5941
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.6
> Environment: AWS EC2 instance
>Reporter: Tomasz Żukowski
>Priority: Minor
>
> When trying to get the connection from MySQLHook below error is raised.
> Airflow version - 1.10.6
> MySQL version - 8.0.15
> MySQLdb (mysqlclient) - tested with 1.3.14(both installed with pip and built
> locally) and 1.4.2.post1
> connection extra:
> {code:python}
> {"charset":"utf8mb4"}
> {code}
> DB charset is set to utf8mb4
> Error message:
> {code:python}
> [2019-11-15 16:55:46,477] {taskinstance.py:1058} ERROR - (2006, "Can't
> initialize character set unknown (path: compiled_in)")
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line
> 930, in _run_raw_task
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 113, in execute
> return_value = self.execute_callable()
> File
> "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py",
> line 118, in execute_callable
> return self.python_callable(*self.op_args, **self.op_kwargs)
> File "", line 136, in mysql_***
> mysql_conn = mysql_hook.get_conn()
> File "/usr/local/lib/python3.7/site-packages/airflow/hooks/mysql_hook.py",
> line 116, in get_conn
> conn = MySQLdb.connect(**conn_config)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/__init__.py",
> line 85, in Connect
> return Connection(*args, **kwargs)
> File
> "/usr/local/lib64/python3.7/site-packages/mysqlclient-1.3.14-py3.7-linux-x86_64.egg/MySQLdb/connections.py",
> line 208, in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> _mysql_exceptions.OperationalError: (2006, "Can't initialize character set
> unknown (path: compiled_in)")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] OmerJog commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator
OmerJog commented on issue #5970: [AIRFLOW-2923][AIRFLOW-1784] Implement LatestOnlyOperator as BaseBranchOperator URL: https://github.com/apache/airflow/pull/5970#issuecomment-554735417 @m1racoli are you working on this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-5582. --- Resolution: Fixed > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > Fix For: 1.10.7 > > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975958#comment-16975958 ] ASF GitHub Bot commented on AIRFLOW-5582: - feluelle commented on pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-5582: -- Fix Version/s: 1.10.7 > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > Fix For: 1.10.7 > > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5582) AutoCommit in jdbc is missing get_autocommit
[ https://issues.apache.org/jira/browse/AIRFLOW-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975959#comment-16975959 ] ASF subversion and git services commented on AIRFLOW-5582: -- Commit d03fb28e69958d8667483d4dbe083fa097ec08e7 in airflow's branch refs/heads/master from Scott Liu [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d03fb28 ] [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232) - add tests - update docs Co-Authored-By: Felix Uellendall > AutoCommit in jdbc is missing get_autocommit > > > Key: AIRFLOW-5582 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5582 > Project: Apache Airflow > Issue Type: Bug > Components: database, hooks >Affects Versions: 1.10.0 >Reporter: Scott Liu >Priority: Major > Fix For: 1.10.7 > > > Some of JDBC driver does not support autoCommit=false, such as > org.apache.hive.jdbc.HiveDrive. > JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, > DbApiHook.get_autocommit retrieve by conn.autocommit. > After DbApiHook.execute(sql), conn.commit() will throws exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle merged pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit
feluelle merged pull request #6232: [AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit URL: https://github.com/apache/airflow/pull/6232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6586: [AIRFLOW-5936] Allow explicit get_pty in SSHOperator
potiuk commented on issue #6586: [AIRFLOW-5936] Allow explicit get_pty in SSHOperator URL: https://github.com/apache/airflow/pull/6586#issuecomment-554727155 Thanks @zzk-dev -> welcome as contributor! The code is merged. Happy that you've learned something and looking forward for future contributions :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5936) Allow explicit get_pty in SSHOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975948#comment-16975948 ] ASF subversion and git services commented on AIRFLOW-5936: -- Commit 150dd9465bc7f304e8de2664565e80cfbef3c55c in airflow's branch refs/heads/master from zzk-dev [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=150dd94 ] [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586) > Allow explicit get_pty in SSHOperator > - > > Key: AIRFLOW-5936 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5936 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.6 >Reporter: Zikun Zhu >Priority: Major > Fix For: 2.0.0 > > > Currently when execution_timeout is reached for an SSHOperator task, the ssh > connection will be closed but the remote process continues to run. In many > scenarios, users might want the process to be killed upon task timeout. > Giving users an explicit get_pty option achieves this goal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5936) Allow explicit get_pty in SSHOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5936. --- Fix Version/s: 2.0.0 Resolution: Fixed > Allow explicit get_pty in SSHOperator > - > > Key: AIRFLOW-5936 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5936 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.6 >Reporter: Zikun Zhu >Priority: Major > Fix For: 2.0.0 > > > Currently when execution_timeout is reached for an SSHOperator task, the ssh > connection will be closed but the remote process continues to run. In many > scenarios, users might want the process to be killed upon task timeout. > Giving users an explicit get_pty option achieves this goal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5936) Allow explicit get_pty in SSHOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975947#comment-16975947 ] ASF GitHub Bot commented on AIRFLOW-5936: - potiuk commented on pull request #6586: [AIRFLOW-5936] Allow explicit get_pty in SSHOperator URL: https://github.com/apache/airflow/pull/6586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow explicit get_pty in SSHOperator > - > > Key: AIRFLOW-5936 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5936 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.6 >Reporter: Zikun Zhu >Priority: Major > > Currently when execution_timeout is reached for an SSHOperator task, the ssh > connection will be closed but the remote process continues to run. In many > scenarios, users might want the process to be killed upon task timeout. > Giving users an explicit get_pty option achieves this goal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6586: [AIRFLOW-5936] Allow explicit get_pty in SSHOperator
potiuk merged pull request #6586: [AIRFLOW-5936] Allow explicit get_pty in SSHOperator URL: https://github.com/apache/airflow/pull/6586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
