seanxwzhang commented on pull request #8545:
URL: https://github.com/apache/airflow/pull/8545#issuecomment-623290340
Addressed most, if not all comments in the previous round of review. Played
a bit with two cases regarding how we fetch DagRuns for SLA consideration:
1. Use a fixed
[
https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098719#comment-17098719
]
ASF GitHub Bot commented on AIRFLOW-249:
seanxwzhang commented on pull request #8545:
URL:
NBardelot opened a new issue #8705:
URL: https://github.com/apache/airflow/issues/8705
The entrypoint manages the two variables `AIRFLOW__CORE__SQL_ALCHEMY_CONN`
and `AIRFLOW__CELERY__BROKER_URL` but does not take into account the fact that
those configurations can be overriden by
boring-cyborg[bot] commented on issue #8705:
URL: https://github.com/apache/airflow/issues/8705#issuecomment-623478219
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an automated
jhtimmins commented on a change in pull request #8681:
URL: https://github.com/apache/airflow/pull/8681#discussion_r419530061
##
File path: airflow/www/views.py
##
@@ -2627,6 +2627,9 @@ class TaskInstanceModelView(AirflowModelView):
'unixname',
jhtimmins commented on pull request #8671:
URL: https://github.com/apache/airflow/pull/8671#issuecomment-623533033
Hi @BasPH @tooptoop4, this PR is a continuation of
https://github.com/apache/airflow/pull/7128, which was started by @sarutak.
This addresses issues causing Airflow to fail
NBardelot opened a new pull request #8708:
URL: https://github.com/apache/airflow/pull/8708
See issue https://github.com/apache/airflow/issues/8706
This is an automated message from the Apache Git Service.
To respond to the
[
https://issues.apache.org/jira/browse/AIRFLOW-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098819#comment-17098819
]
Timur Rubeko edited comment on AIRFLOW-5506 at 5/4/20, 10:00 AM:
-
I'm
boring-cyborg[bot] commented on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623383306
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an automated
turbaszek opened a new issue #8703:
URL: https://github.com/apache/airflow/issues/8703
**Description**
Add support for `set` in Xcom serialization method. This requires support
for `set` in nested structures.
**Use case / motivation**
Once we switched to JSON as
kaxil commented on a change in pull request #8652:
URL: https://github.com/apache/airflow/pull/8652#discussion_r419341525
##
File path: scripts/ci/pylint_todo.txt
##
@@ -24,6 +24,7 @@
./airflow/models/taskinstance.py
./airflow/models/variable.py
./airflow/models/xcom.py
turbaszek commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419341652
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
turbaszek commented on a change in pull request #8652:
URL: https://github.com/apache/airflow/pull/8652#discussion_r419349353
##
File path: airflow/models/xcom_arg.py
##
@@ -0,0 +1,147 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor
jhtimmins commented on a change in pull request #8681:
URL: https://github.com/apache/airflow/pull/8681#discussion_r419530061
##
File path: airflow/www/views.py
##
@@ -2627,6 +2627,9 @@ class TaskInstanceModelView(AirflowModelView):
'unixname',
NBardelot opened a new pull request #8707:
URL: https://github.com/apache/airflow/pull/8707
See issue https://github.com/apache/airflow/issues/8705
This is an automated message from the Apache Git Service.
To respond to the
NBardelot opened a new issue #8706:
URL: https://github.com/apache/airflow/issues/8706
It is a common good practice for Docker/Kubernetes to create a user who is
not root to run the main process of the container of course, but it is also a
good practice to make the user member of the root
anitakar opened a new pull request #8709:
URL: https://github.com/apache/airflow/pull/8709
Support concurrent migrations in alembic.
Can help when deployment is scripted and consists of multiple components
The solution is based on
CodingJonas commented on pull request #8617:
URL: https://github.com/apache/airflow/pull/8617#issuecomment-623339870
The Pipeline fails on an unrelated test:
```
> self.assertEqual(State.RUNNING, ti.state)
E AssertionError: 'running' != 'success'
E - running
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098794#comment-17098794
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
feluelle commented on pull request #7030:
URL:
[
https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098795#comment-17098795
]
ASF GitHub Bot commented on AIRFLOW-6440:
-
feluelle edited a comment on pull request #7030:
feluelle commented on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-623350479
> For now, I'll make this Executor an optional plugin for Airflow. Thank you
all for contributing and your suggestions.
Do I understand it correctly that you want to
feluelle edited a comment on pull request #7030:
URL: https://github.com/apache/airflow/pull/7030#issuecomment-623350479
> For now, I'll make this Executor an optional plugin for Airflow. Thank you
all for contributing and your suggestions.
@aelzeiny Do I understand it correctly
mik-laj commented on a change in pull request #8618:
URL: https://github.com/apache/airflow/pull/8618#discussion_r419324418
##
File path: tests/providers/amazon/aws/operators/test_emr_system.py
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
kaxil commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419339392
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
mik-laj opened a new pull request #8704:
URL: https://github.com/apache/airflow/pull/8704
Backport: https://github.com/apache/airflow/pull/8290
---
Make sure to mark the boxes below before creating PR: [x]
- [X] Description above provides context of the change
- [X] Unit
turbaszek commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419338031
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
kaxil commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419338913
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
michalslowikowski00 commented on pull request #8700:
URL: https://github.com/apache/airflow/pull/8700#issuecomment-623415047
Great as always
This is an automated message from the Apache Git Service.
To respond to the
feluelle commented on a change in pull request #8618:
URL: https://github.com/apache/airflow/pull/8618#discussion_r419298191
##
File path: tests/providers/amazon/aws/operators/test_emr_system.py
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
boring-cyborg[bot] removed a comment on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623383306
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an
turbaszek commented on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623384203
As I said in
https://github.com/apache/airflow/pull/8607#discussion_r419327697
> I was thinking about that and I am against it. JSON has no set structure
and users should
fuyi commented on pull request #8675:
URL: https://github.com/apache/airflow/pull/8675#issuecomment-623418565
Hi @jeffolsi, Thanks for the comment. I agree with you that "invalid date"
is not correct either. I have talked with @XD-DENG and leave this out of the
scope of this PR.
mik-laj commented on pull request #8700:
URL: https://github.com/apache/airflow/pull/8700#issuecomment-623364635
CC: @michalslowikowski00 Can I ask for review?
This is an automated message from the Apache Git Service.
To
[
https://issues.apache.org/jira/browse/AIRFLOW-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098819#comment-17098819
]
Timur Rubeko commented on AIRFLOW-5506:
---
I'm not sure if all the comments here relate to the same
turbaszek commented on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623388643
> We do it anyway for Serialized DAGs
And if I am not mistaken we make an assumption that we will never reach max
concurrency error?
> I am surprised though that we
turbaszek commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419329701
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
feluelle commented on a change in pull request #8618:
URL: https://github.com/apache/airflow/pull/8618#discussion_r419299835
##
File path: tests/providers/amazon/aws/operators/test_emr_system.py
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
boring-cyborg[bot] commented on pull request #8701:
URL: https://github.com/apache/airflow/pull/8701#issuecomment-623378171
Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our
VijayantSoni opened a new pull request #8701:
URL: https://github.com/apache/airflow/pull/8701
Adding ElastiCache Hook for creating, describing and deleting replication
groups.
This hook interacts with AWS ElastiCache and supports following operations:
1. Creating replication
kaxil commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419336331
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
Acehaidrey opened a new pull request #8702:
URL: https://github.com/apache/airflow/pull/8702
Add context to execution_date_fn in ExternalTaskSensor
Will add details later today -
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above
turbaszek commented on a change in pull request #8652:
URL: https://github.com/apache/airflow/pull/8652#discussion_r419345843
##
File path: scripts/ci/pylint_todo.txt
##
@@ -24,6 +24,7 @@
./airflow/models/taskinstance.py
./airflow/models/variable.py
./airflow/models/xcom.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098778#comment-17098778
]
ASF GitHub Bot commented on AIRFLOW-4549:
-
TeddyHartanto commented on pull request #7735:
URL:
TeddyHartanto commented on pull request #7735:
URL: https://github.com/apache/airflow/pull/7735#issuecomment-623337588
FYI: this PR deprecates the outdated PR #5308
This is an automated message from the Apache Git Service.
mik-laj opened a new pull request #8700:
URL: https://github.com/apache/airflow/pull/8700
Fix: https://github.com/apache/airflow/issues/8659
---
Make sure to mark the boxes below before creating PR: [x]
- [X] Description above provides context of the change
- [X] Unit tests
kaxil commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419327697
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
turbaszek commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419330497
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
turbaszek commented on a change in pull request #8607:
URL: https://github.com/apache/airflow/pull/8607#discussion_r419338031
##
File path: airflow/operators/latest_only_operator.py
##
@@ -46,7 +46,7 @@ def choose_branch(self, context: Dict) -> Union[str,
Iterable[str]]:
kaxil commented on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623385925
We do it anyway for Serialized DAGs so we can just reuse the following code
that takes care of set and other nested data structures:
kaxil commented on issue #8703:
URL: https://github.com/apache/airflow/issues/8703#issuecomment-623386519
I am surprised though that we didn't get any issues regarding it yet (or
maybe we just haven't looked) :D
This is an
feluelle commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-623628892
> I see that you have much bigger problems, I would like to suggest a small
change, about naming "aws_system_helpers.py" as "amazon_system_helpers.py"
since it contains
jlazarevicm commented on issue #8507:
URL: https://github.com/apache/airflow/issues/8507#issuecomment-623627672
@jeffolsi I think you are not right. Event though if we set
`get_all_pages=False` it doesn't change the fact that this line (prior than
checking if `get_all_pages` is set to
tedmiston opened a new pull request #8710:
URL: https://github.com/apache/airflow/pull/8710
I noticed a few places in the codebase were trailing list commas were used
inconsistently. This PR is an enhancement to improve the consistency of list
trailing comma usage.
More
mik-laj edited a comment on pull request #8710:
URL: https://github.com/apache/airflow/pull/8710#issuecomment-623633477
How do you want to prevent regression?Without automatic checks, it is very
difficult to watch the style in the OSS project.
jlazarevicm edited a comment on issue #8507:
URL: https://github.com/apache/airflow/issues/8507#issuecomment-623627672
@potiuk, @jeffolsi I think you are not right. Event though if we set
`get_all_pages=False` it doesn't change the fact that this line (prior than
checking if
Acehaidrey commented on a change in pull request #8681:
URL: https://github.com/apache/airflow/pull/8681#discussion_r419658098
##
File path: airflow/www/views.py
##
@@ -2627,6 +2627,9 @@ class TaskInstanceModelView(AirflowModelView):
'unixname',
jlazarevicm edited a comment on issue #8507:
URL: https://github.com/apache/airflow/issues/8507#issuecomment-623627672
@potiuk, @jeffolsi I think you are not right. Event though if we set
`get_all_pages=False` it doesn't change the fact that this line (prior than
checking if
jaketf commented on pull request #7197:
URL: https://github.com/apache/airflow/pull/7197#issuecomment-623613676
@mik-laj thanks it does make more sense in that module.
looks like those docs don't really touch on sensors so `poke_mode_only` docs
there would've seem out of place.
[
https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099186#comment-17099186
]
ASF GitHub Bot commented on AIRFLOW-6586:
-
jaketf commented on pull request #7197:
URL:
mik-laj commented on pull request #8710:
URL: https://github.com/apache/airflow/pull/8710#issuecomment-623633477
How do you want to prevent regression?
This is an automated message from the Apache Git Service.
To respond to
mik-laj edited a comment on pull request #8710:
URL: https://github.com/apache/airflow/pull/8710#issuecomment-623633477
How do you want to prevent regression?Without automatic checks, it is very
difficult to keep one style in the OSS project.
msgongora commented on issue #25:
URL:
https://github.com/apache/airflow-on-k8s-operator/issues/25#issuecomment-623707260
this is a kustomize bug violating [yaml standard
(---)](https://yaml.org/spec/1.2/spec.html#id2760395). I've submitted a
jhtimmins commented on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623732850
@ashb No, it comes up in (at least) 3.7.7 as well.
This is an automated message from the Apache Git Service.
tedmiston commented on pull request #8710:
URL: https://github.com/apache/airflow/pull/8710#issuecomment-623655524
@mik-laj I agree it's difficult to keep consistent styles longterm without
automatic checking.
When I'm working on a new PR changing code that doesn't have an automatic
kaxil commented on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623738082
Similar issues:
- https://github.com/docker-library/python/issues/461
- https://github.com/awslabs/aws-data-wrangler/issues/151
Acehaidrey commented on pull request #8702:
URL: https://github.com/apache/airflow/pull/8702#issuecomment-623739358
@jhtimmins if you don't mind when you get a chance to look at some of this
and share your thoughts as well
mik-laj commented on a change in pull request #7888:
URL: https://github.com/apache/airflow/pull/7888#discussion_r419719858
##
File path: airflow/utils/dag_processing.py
##
@@ -559,7 +560,7 @@ def __init__(self,
AbstractDagFileProcessorProcess
ashb commented on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623729015
This only affects Py3.8+ right?
This is an automated message from the Apache Git Service.
To respond to the
mik-laj commented on pull request #8693:
URL: https://github.com/apache/airflow/pull/8693#issuecomment-623731839
@kaxil I updated the description
This is an automated message from the Apache Git Service.
To respond to the
Acehaidrey commented on pull request #8681:
URL: https://github.com/apache/airflow/pull/8681#issuecomment-623731909
Thank you team!
This is an automated message from the Apache Git Service.
To respond to the message, please
kaxil commented on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623736554
Makes me wonder, what changed !!
This is an automated message from the Apache Git Service.
To respond to the
jhtimmins commented on pull request #8702:
URL: https://github.com/apache/airflow/pull/8702#issuecomment-623752148
@Acehaidrey Happy to take a look!
This is an automated message from the Apache Git Service.
To respond to the
nadflinn commented on issue #8480:
URL: https://github.com/apache/airflow/issues/8480#issuecomment-623720953
Following up on this, it's not ideal, but if in 1.10.9 you want to set a
specific worker process count to, for example, 8 workers then just include:
`worker_autoscale = 8,8`
jhtimmins opened a new pull request #8711:
URL: https://github.com/apache/airflow/pull/8711
Currently, dag_processing.py imports the `multiprocessing` library, and uses
`multiprocessing.connection.Connection` to declare a `Connection` type for the
`DagFileProcessorManager` class. This
kaxil edited a comment on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623737050
Most likely was introduced in **Python 3.7.7**
This is an automated message from the Apache Git Service.
kaxil commented on pull request #8711:
URL: https://github.com/apache/airflow/pull/8711#issuecomment-623737050
Most likely was introduced in Python 3.7
This is an automated message from the Apache Git Service.
To respond to
boring-cyborg[bot] commented on pull request #8681:
URL: https://github.com/apache/airflow/pull/8681#issuecomment-623730693
Awesome work, congrats on your first merged pull request!
This is an automated message from the
boittega commented on a change in pull request #8305:
URL: https://github.com/apache/airflow/pull/8305#discussion_r419776461
##
File path: docs/howto/operator/apache/spark.rst
##
@@ -0,0 +1,102 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more
kaxil commented on pull request #8693:
URL: https://github.com/apache/airflow/pull/8693#issuecomment-623726700
Can you add a description to the PR with before and After or why this is
needed
This is an automated message
Acehaidrey commented on pull request #8680:
URL: https://github.com/apache/airflow/pull/8680#issuecomment-623739563
@jhtimmins if you don't mind when you get a chance to look at some of this
and share your thoughts as well
dimberman commented on pull request #8702:
URL: https://github.com/apache/airflow/pull/8702#issuecomment-623796304
@Acehaidrey I agree with @jhtimmins. I think having one function is cleaner
and will make it easier if we decide to deprecate one of the methods later.
boring-cyborg[bot] commented on issue #8712:
URL: https://github.com/apache/airflow/issues/8712#issuecomment-623761225
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an automated
ishioni opened a new issue #8712:
URL: https://github.com/apache/airflow/issues/8712
**Apache Airflow version**: 1.10.10
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`): 1.15.11
**Environment**: dockerhub image
Acehaidrey commented on pull request #8702:
URL: https://github.com/apache/airflow/pull/8702#issuecomment-623759382
@jhtimmins thank you so much :) I know I've been taking a lot of your time
these days so appreciate it!
boring-cyborg[bot] commented on issue #8713:
URL: https://github.com/apache/airflow/issues/8713#issuecomment-623761771
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an automated
boittega opened a new issue #8713:
URL: https://github.com/apache/airflow/issues/8713
**Apache Airflow version**: 1.10.10
**What happened**:
`SparkSqlHook` is not using any connection, the default conn_id is
`spark_sql_default`, if this connection doesn't exist, the hook
jhtimmins commented on pull request #8702:
URL: https://github.com/apache/airflow/pull/8702#issuecomment-623787422
@Acehaidrey Of course :) Always happy to take a look.
I think the use case makes sense, but the implementation could be made more
explicit.
Personally, I think
jhtimmins commented on a change in pull request #8680:
URL: https://github.com/apache/airflow/pull/8680#discussion_r419825461
##
File path: airflow/models/taskinstance.py
##
@@ -1080,6 +1081,8 @@ def signal_handler(signum, frame):
self.handle_failure(e, test_mode,
bhavaniravi opened a new issue #8714:
URL: https://github.com/apache/airflow/issues/8714
**Description**
KubernetesExecutor/KubernetesPodOperator supports `securityContext` for pods
but I am working on a case where I want to set a `privileged` mode for base
container.
**Use
boring-cyborg[bot] commented on issue #8714:
URL: https://github.com/apache/airflow/issues/8714#issuecomment-623868717
Thanks for opening your first issue here! Be sure to follow the issue
template!
This is an automated
turbaszek commented on a change in pull request #8652:
URL: https://github.com/apache/airflow/pull/8652#discussion_r419257234
##
File path: scripts/ci/pylint_todo.txt
##
@@ -24,6 +24,7 @@
./airflow/models/taskinstance.py
./airflow/models/variable.py
./airflow/models/xcom.py
turbaszek commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-623308491
Hi @exploy , you can use `table_resource` argument to pass any table
definition you want, no need to specify every single parameter.
michalslowikowski00 commented on issue #8659:
URL: https://github.com/apache/airflow/issues/8659#issuecomment-623309946
@mik-laj I know how to fix it. I didn't have time for it. :(
This is an automated message from the
jeffolsi commented on issue #8649:
URL: https://github.com/apache/airflow/issues/8649#issuecomment-623311926
@BasPH I'm running 1.10.3
I'm not sure what exactly to report on the new issue. I don't consider this
a bug but maybe i'm wrong. I just wanted to explain why the suggestion to
94 matches
Mail list logo
