date:20200310

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390754114
 
 

 ##
 File path: tests/providers/amazon/aws/operators/test_redshift_to_s3.py
 ##
 @@ -76,3 +76,49 @@ def test_execute(self, table_as_file_name, expected_s3_key, 
mock_run, mock_sessi
 
 assert mock_run.call_count == 1
 assert_equal_ignore_multiple_spaces(self, mock_run.call_args[0][0], 
unload_query)
+
+@parameterized.expand([
+[True, "key/None_"],
+[False, "key"],
 
 Review comment:
   I like that you used parameterized. But we need to add custom_select_query 
to parameterized parameters and also we need to handle the failure cases here 
(with various combinations of None parameters set - just table None, just 
schema None, all None etc so the parameterized should have the "expected 
exception" also (or maybe we should have failure test cases as separate method 
- both will work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390753208
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if not self.custom_select_query:
+if (not self.schema) or (not self.table):
+raise AirflowBadRequest("schema, table set to None")
+select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+else:
+select_query = self.custom_select_query
 
 Review comment:
   ```suggestion
   if all([self.schema, self.table]):
 select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
   elif self.custom_select_query:
 select_query = self.custom_select_query
   else:
raise AirflowBadRequest(f"Either (schema, table) or 
custom_select_query should be set. They are ({self.schema},{self.table}) and 
{self.custom_select_query} now.")
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390753368
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if not self.custom_select_query:
+if (not self.schema) or (not self.table):
+raise AirflowBadRequest("schema, table set to None")
+select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+else:
+select_query = self.custom_select_query
 
 Review comment:
   That would be more pythonic.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390751106
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1047,6 +1047,20 @@ If the DAG relies on tasks with other trigger rules 
(i.e. `all_done`) being skip
 
 The goal of this change is to achieve a more consistent and configurale 
cascading behaviour based on the `BaseBranchOperator` (see 
[AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and 
[AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
 
+
+### RedshiftToS3Transfer:: signature changed
+
+Previous versions of the `RedshiftToS3Transfer` operator required `schema` and 
`table` arguments as the first 2
+positional arguments. This signature was changed in 2.0 and
+the `s3_bucket` and `s3_key` are the first 2 positional reguements.
+
+In order to use this operator:
+```python
+result = RedshiftToS3Transfer('schema', 'table')  # Pre-2.0 call
+...
+result = RedshiftToS3Transfer('s3_bucket', 's3_key')  # Post-2.0 call
 
 Review comment:
   I think the examples are not correct and it might be misleading - the first 
one should also have s3_bucket and s3_key -> they were not optional. Also the 
second example is not valid because it requires either schema+table combination 
or custom query.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390752987
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
+if self.schema is None or self.table is None:
+raise AirflowBadRequest("schema, table set to None")
 
 Review comment:
   @swiftomkar WDYT, about this one?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7683: [AIRFLOW-7033] Change dag and task state meanwhile

zhongjiajie commented on a change in pull request #7683: [AIRFLOW-7033] Change 
dag and task state meanwhile
URL: https://github.com/apache/airflow/pull/7683#discussion_r390752331
 
 

 ##
 File path: airflow/api/common/experimental/mark_tasks.py
 ##
 @@ -340,19 +400,10 @@ def set_dag_run_state_to_failed(dag, execution_date, 
commit=False, session=None)
 _set_dag_run_state(dag.dag_id, execution_date, State.FAILED, session)
 
 # Mark only RUNNING task instances.
-task_ids = [task.task_id for task in dag.tasks]
-tis = session.query(TaskInstance).filter(
+tasks = session.query(TaskInstance).filter(
 TaskInstance.dag_id == dag.dag_id,
-TaskInstance.execution_date == execution_date,
-TaskInstance.task_id.in_(task_ids)).filter(TaskInstance.state == 
State.RUNNING)
-task_ids_of_running_tis = [task_instance.task_id for task_instance in tis]
-
-tasks = []
-for task in dag.tasks:
-if task.task_id not in task_ids_of_running_tis:
-continue
-task.dag = dag
-tasks.append(task)
 
 Review comment:
   And also this one?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7683: [AIRFLOW-7033] Change dag and task state meanwhile

zhongjiajie commented on a change in pull request #7683: [AIRFLOW-7033] Change 
dag and task state meanwhile
URL: https://github.com/apache/airflow/pull/7683#discussion_r390752240
 
 

 ##
 File path: airflow/api/common/experimental/mark_tasks.py
 ##
 @@ -312,8 +374,6 @@ def set_dag_run_state_to_success(dag, execution_date, 
commit=False, session=None
 _set_dag_run_state(dag.dag_id, execution_date, State.SUCCESS, session)
 
 # Mark all task instances of the dag run to success.
-for task in dag.tasks:
-task.dag = dag
 
 Review comment:
   Could we remove this code?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

codecov-io commented on issue #7688: [AIRFLOW-6794] Allow AWS Operator 
RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#issuecomment-597447171
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=h1) 
Report
   > Merging 
[#7688](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/85465499074443fa98322051ca11f43c66242cc3=desc)
 will **decrease** coverage by `0.17%`.
   > The diff coverage is `85.71%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7688/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7688  +/-   ##
   ==
   - Coverage   86.83%   86.66%   -0.18% 
   ==
 Files 897  904   +7 
 Lines   4280343742 +939 
   ==
   + Hits3717037909 +739 
   - Misses   5633 5833 +200 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...w/providers/amazon/aws/operators/redshift\_to\_s3.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvcmVkc2hpZnRfdG9fczMucHk=)
 | `95.34% <85.71%> (-1.95%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==)
 | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | 
[airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==)
 | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
   | 
[airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==)
 | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | 
[...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==)
 | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
   | 
[airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5)
 | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
   | 
[...rflow/providers/google/cloud/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9kYXRhZmxvdy5weQ==)
 | `90.44% <0.00%> (-8.65%)` | :arrow_down: |
   | 
[airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=)
 | `90.53% <0.00%> (-0.12%)` | :arrow_down: |
   | 
[airflow/providers/mysql/hooks/mysql.py](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvaG9va3MvbXlzcWwucHk=)
 | `92.70% <0.00%> (-0.07%)` | :arrow_down: |
   | ... and [21 
more](https://codecov.io/gh/apache/airflow/pull/7688/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=footer). 
Last update 
[8546549...98f875f](https://codecov.io/gh/apache/airflow/pull/7688?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390719168
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1047,6 +1048,21 @@ If the DAG relies on tasks with other trigger rules 
(i.e. `all_done`) being skip
 
 The goal of this change is to achieve a more consistent and configurale 
cascading behaviour based on the `BaseBranchOperator` (see 
[AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and 
[AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
 
+## Airflow 2.0
 
 Review comment:
   @potiuk Got it! Thanks! Can you take a look to see if this looks good?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] swiftomkar commented on issue #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar commented on issue #7688: [AIRFLOW-6794] Allow AWS Operator 
RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#issuecomment-597417616
 
 
   @zhongjiajie Thanks for your suggestions!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390719168
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1047,6 +1048,21 @@ If the DAG relies on tasks with other trigger rules 
(i.e. `all_done`) being skip
 
 The goal of this change is to achieve a more consistent and configurale 
cascading behaviour based on the `BaseBranchOperator` (see 
[AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and 
[AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
 
+## Airflow 2.0
 
 Review comment:
   @potiuk Got it! Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7685: [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op

zhongjiajie commented on a change in pull request #7685: [AIRFLOW-7034] Remove 
feature: Assigning Dag to task using Bitshift Op
URL: https://github.com/apache/airflow/pull/7685#discussion_r390711422
 
 

 ##
 File path: airflow/providers/google/cloud/operators/pubsub.py
 ##
 @@ -196,8 +196,7 @@ class PubSubCreateSubscriptionOperator(BaseOperator):
 
 with DAG('failing DAG') as dag:
 (
-dag
->> PubSubSubscriptionCreateOperator(
+PubSubSubscriptionCreateOperator(
 
 Review comment:
   FYI, still have this syntax in line 44, 56, 185, 373, 382, 485, 496


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390707943
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1047,6 +1048,21 @@ If the DAG relies on tasks with other trigger rules 
(i.e. `all_done`) being skip
 
 The goal of this change is to achieve a more consistent and configurale 
cascading behaviour based on the `BaseBranchOperator` (see 
[AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and 
[AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
 
+## Airflow 2.0
 
 Review comment:
   BTW, should we put the latest change in the top of UPDATING.md @potiuk 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390705298
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
+if self.schema is None or self.table is None:
 
 Review comment:
   ```suggestion
   if (not self.schema) or (not self.table):
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390705128
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
 
 Review comment:
   ```suggestion
   if not self.custom_select_query:
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

zhongjiajie commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390707156
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
+if self.schema is None or self.table is None:
+raise AirflowBadRequest("schema, table set to None")
 
 Review comment:
   I think is better to use `Both custom_select_query and (schema, table) be 
None not allow here` or some else, you could not just hint with `schema` and 
`table`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on issue #7683: [AIRFLOW-7033] Change dag and task state meanwhile

zhongjiajie commented on issue #7683: [AIRFLOW-7033] Change dag and task state 
meanwhile
URL: https://github.com/apache/airflow/pull/7683#issuecomment-597399419
 
 
   Also and make_success/make_failed to dag.html/tree.html
   ![](https://i.loli.net/2020/03/11/FLHtAsecoO4ZDlp.png)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-6859) Airflow scheduler looking for logs of example DAGs in the wrong folder

2020-03-10 Thread Abhilash Kishore (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhilash Kishore closed AIRFLOW-6859.
-
Resolution: Invalid

> Airflow scheduler looking for logs of example DAGs in the wrong folder
> --
>
> Key: AIRFLOW-6859
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6859
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.9
>Reporter: Abhilash Kishore
>Priority: Minor
>
> My AIRFLOW_HOME is set to */home/airflow*
> I am running Airflow as a service
> `cat /etc/systemd/system/airflow-scheduler.service` :
>  
> {code:java}
> [Unit]
> Description=Airflow scheduler daemon
> After=network.target postgresql.service mysql.service redis.service 
> rabbitmq-server.service mssql-server.service
> Wants=postgresql.service mysql.service redis.service rabbitmq-server.service 
> mssql-server.service
> [Service]
> EnvironmentFile=/etc/sysconfig/airflow
> User=airflow
> Group=airflow
> Type=simple
> ExecStart=/usr/local/bin/airflow scheduler
> Restart=always
> RestartSec=5s
> [Install]
> WantedBy=multi-user.target
> {code}
>  
>  
> *BUG:*
> Airflow scheduler looking for logs of example DAGs in the wrong folder. 
> Example:
>  
> {code:java}
> Feb 21 01:57:24 airflow-poc airflow[25847]: Process 
> DagFileProcessor281-Process:
> Feb 21 01:57:24 airflow-poc airflow[25847]: Traceback (most recent call last):
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
> Feb 21 01:57:24 airflow-poc airflow[25847]: self.run()
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
> Feb 21 01:57:24 airflow-poc airflow[25847]: self._target(*self._args, 
> **self._kwargs)
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 
> 135, in _run_file_processor
> Feb 21 01:57:24 airflow-poc airflow[25847]: set_context(log, file_path)
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/logging_mixin.py", 
> line 198, in set_context
> Feb 21 01:57:24 airflow-poc airflow[25847]: handler.set_context(value)
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/local/lib/python3.6/dist-packages/airflow/utils/log/file_processor_handler.py",
>  line 66, in set_context
> Feb 21 01:57:24 airflow-poc airflow[25847]: self.handler = 
> logging.FileHandler(local_loc)
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/lib/python3.6/logging/__init__.py", line 1032, in __init__
> Feb 21 01:57:24 airflow-poc airflow[25847]: StreamHandler.__init__(self, 
> self._open())
> Feb 21 01:57:24 airflow-poc airflow[25847]:   File 
> "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
> Feb 21 01:57:24 airflow-poc airflow[25847]: return 
> open(self.baseFilename, self.mode, encoding=self.encoding)
> Feb 21 01:57:24 airflow-poc airflow[25847]: PermissionError: [Errno 13] 
> Permission denied: 
> '/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log'{code}
>  
>  
> The same is happening for all example DAGs (haven't loaded any other DAGs 
> yet).
>  
> Airflow scheduler is looking for the logs of the example DAG in 
> `/home/airflow/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`.
>  It should instead look in 
> `/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py.log`
>   instead (without the '/home/airflow' prefix).
>  
> In the `dag` table in the database, the DAG with dag_id `example_skip_dag.` 
> has fileloc 
> `/usr/local/lib/python3.6/dist-packages/airflow/example_dags/example_skip_dag.py`.
>  
> Not sure why Airflow scheduler is appending `/home/airflow' to this path
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-7039) Specific DAG Schedule & DST Results in Skipped DAG Run

2020-03-10 Thread Peter Kim (Jira)

Peter Kim created AIRFLOW-7039:
--

 Summary: Specific DAG Schedule & DST Results in Skipped DAG Run
 Key: AIRFLOW-7039
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7039
 Project: Apache Airflow
  Issue Type: Bug
  Components: scheduler
Affects Versions: 1.10.7
 Environment: Amazon Linux 2 AMI
Reporter: Peter Kim


*Scenario:* 
EC2 running airflow is in Eastern Time (America/New_York), 
airflow.cfg>[core]>default_timezone=America/New_York (automatically changes 
correctly)

Monday morning after Daylight Savings Time applied a handful of DAG runs were 
not executed as expected.  The strange part is that these DAGs were the only 
jobs that did not behave as expected, all other DAGs ran normally.  
Additionally, only the first expected run after DST was skipped, subsequent 
runs later that day were scheduled successfully.

Here is the pattern observed:

DAG Schedule which skipped first run:  (0 , * * 1,2,3,4,5)
e.g. Schedules M-F, with two distinct runs per day.
DAGs that run at one time, M-F & DAGs that run at two times, not M-F did not 
experience this issue.  

 

Based on the logs, it appears as if the expected run that was missed was not 
seen by the scheduler whatsoever (see below):

 

 

2020 03 06 6:30 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
[2020-03-06 06:31:01,220] \{logging_mixin.py:112} INFO - [2020-03-06 
06:31:01,220] \{settings.py:254} INFO - settings.configure_orm(): Using pool 
settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=697
[2020-03-06 06:31:01,222] \{scheduler_job.py:153} INFO - Started process 
(PID=697) to work on /home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 06:31:01,228] \{scheduler_job.py:1539} INFO - Processing file 
/home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
[2020-03-06 06:31:01,228] \{logging_mixin.py:112} INFO - [2020-03-06 
06:31:01,228] \{dagbag.py:403} INFO - Filling up the DagBag from 
/home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 06:31:01,238] \{scheduler_job.py:1551} INFO - DAG(s) 
dict_keys(['']) retrieved from 
/home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 06:31:01,305] \{scheduler_job.py:1262} INFO - Processing 
[2020-03-06 06:31:01,348] \{logging_mixin.py:112} WARNING - 
/home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170: 
Warning: (1300, "Invalid utf8mb4 character string: '80047D'") result = 
self._query(query)
[2020-03-06 06:31:01,362] \{scheduler_job.py:1272} INFO - Created  @ 2020-03-05T15:30:00+00:00: scheduled__2020-03-05T15:30:00+00:00, 
externally triggered: False>
[2020-03-06 06:31:01,366] \{scheduler_job.py:740} INFO - Examining DAG run 
 @ 2020-03-05 15:30:00+00:00: 
scheduled__2020-03-05T15:30:00+00:00, externally triggered: False>
[2020-03-06 06:31:01,389] \{scheduler_job.py:440} INFO - Skipping SLA check for 
> because no tasks in DAG have SLAs
[2020-03-06 06:31:01,395] \{scheduler_job.py:1613} INFO - Creating / updating 
. 2020-03-05 15:30:00+00:00 [scheduled]> in 
ORM
[2020-03-06 06:31:01,414] \{scheduler_job.py:161} INFO - Processing 
/home/ec2-user/airflow/s3fuse/dags/.py took 0.192 seconds
20200306 10 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
[2020-03-06 10:30:00,083] \{logging_mixin.py:112} INFO - [2020-03-06 
10:30:00,082] \{settings.py:254} INFO - settings.configure_orm(): Using pool 
settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=16194
[2020-03-06 10:30:00,085] \{scheduler_job.py:153} INFO - Started process 
(PID=16194) to work on /home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 10:30:00,090] \{scheduler_job.py:1539} INFO - Processing file 
/home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
[2020-03-06 10:30:00,090] \{logging_mixin.py:112} INFO - [2020-03-06 
10:30:00,090] \{dagbag.py:403} INFO - Filling up the DagBag from 
/home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 10:30:00,099] \{scheduler_job.py:1551} INFO - DAG(s) 
dict_keys(['']) retrieved from 
/home/ec2-user/airflow/s3fuse/dags/.py
[2020-03-06 10:30:00,159] \{scheduler_job.py:1262} INFO - Processing 
[2020-03-06 10:30:00,193] \{logging_mixin.py:112} WARNING - 
/home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170: 
Warning: (1300, "Invalid utf8mb4 character string: '80047D'")
  result = self._query(query)
[2020-03-06 10:30:00,207] \{scheduler_job.py:1272} INFO - Created  @ 2020-03-06T11:30:00+00:00: scheduled__2020-03-06T11:30:00+00:00, 
externally triggered: False>
[2020-03-06 10:30:00,212] \{scheduler_job.py:740} INFO - Examining DAG run 
 @ 2020-03-06 11:30:00+00:00: 
scheduled__2020-03-06T11:30:00+00:00, externally triggered: False>
[2020-03-06 10:30:00,232] \{scheduler_job.py:440} INFO - Skipping SLA check for 
> because no tasks in DAG have SLAs
[2020-03-06 10:30:00,236] \{scheduler_job.py:1613} INFO - Creating / updating 
. 2020-03-06 11:30:00+00:00 [scheduled]> in 
ORM
[2020-03-06 10:30:00,251] \{scheduler_job.py:161} INFO -

[GitHub] [airflow] potiuk commented on issue #7686: New user company

potiuk commented on issue #7686: New user company
URL: https://github.com/apache/airflow/pull/7686#issuecomment-597348015
 
 
   Can you please correct the commit title ?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pingzh commented on a change in pull request #7490: [AIRFLOW-6860] Default ignore_first_depends_on_past to True

pingzh commented on a change in pull request #7490: [AIRFLOW-6860] Default 
ignore_first_depends_on_past to True
URL: https://github.com/apache/airflow/pull/7490#discussion_r390643580
 
 

 ##
 File path: airflow/cli/commands/dag_command.py
 ##
 @@ -67,6 +67,13 @@ def dag_backfill(args, dag=None):
 
 signal.signal(signal.SIGTERM, sigint_handler)
 
+import warnings
+warnings.warn('--ignore_first_depends_on_past is deprecated as the value 
is always set to True',
 
 Review comment:
   thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

potiuk commented on issue #7656: [AIRFLOW-7013] Automated check if Breeze image 
needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#issuecomment-597332080
 
 
   All should be good now!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390623978
 
 

 ##
 File path: UPDATING.md
 ##
 @@ -1047,6 +1048,21 @@ If the DAG relies on tasks with other trigger rules 
(i.e. `all_done`) being skip
 
 The goal of this change is to achieve a more consistent and configurale 
cascading behaviour based on the `BaseBranchOperator` (see 
[AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and 
[AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
 
+## Airflow 2.0
 
 Review comment:
   It's actually Airflow Master :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if 
Breeze image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#issuecomment-597323139
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=h1) 
Report
   > Merging 
[#7656](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fcfc2aa7317916b402340e223b4431f87d9d6d6f=desc)
 will **decrease** coverage by `0.48%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7656/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7656  +/-   ##
   ==
   - Coverage   86.98%   86.49%   -0.49% 
   ==
 Files 904  904  
 Lines   4373643736  
   ==
   - Hits3804537831 -214 
   - Misses   5691 5905 +214 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0.00%> (-72.16%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
   | 
[airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==)
 | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=)
 | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
   | 
[...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=)
 | `52.94% <0.00%> (-32.36%)` | :arrow_down: |
   | 
[airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=)
 | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=footer). 
Last update 
[fcfc2aa...115badc](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if 
Breeze image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#issuecomment-597323139
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=h1) 
Report
   > Merging 
[#7656](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fcfc2aa7317916b402340e223b4431f87d9d6d6f=desc)
 will **decrease** coverage by `0.48%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7656/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7656  +/-   ##
   ==
   - Coverage   86.98%   86.49%   -0.49% 
   ==
 Files 904  904  
 Lines   4373643736  
   ==
   - Hits3804537831 -214 
   - Misses   5691 5905 +214 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0.00%> (-72.16%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
   | 
[airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==)
 | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=)
 | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
   | 
[...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=)
 | `52.94% <0.00%> (-32.36%)` | :arrow_down: |
   | 
[airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=)
 | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=footer). 
Last update 
[fcfc2aa...115badc](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

codecov-io edited a comment on issue #7656: [AIRFLOW-7013] Automated check if 
Breeze image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#issuecomment-597323139
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=h1) 
Report
   > Merging 
[#7656](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fcfc2aa7317916b402340e223b4431f87d9d6d6f=desc)
 will **decrease** coverage by `0.48%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7656/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7656  +/-   ##
   ==
   - Coverage   86.98%   86.49%   -0.49% 
   ==
 Files 904  904  
 Lines   4373643736  
   ==
   - Hits3804537831 -214 
   - Misses   5691 5905 +214 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0.00%> (-72.16%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
   | 
[airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==)
 | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=)
 | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
   | 
[...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=)
 | `52.94% <0.00%> (-32.36%)` | :arrow_down: |
   | 
[airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=)
 | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
   | ... and [6 
more](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=footer). 
Last update 
[fcfc2aa...115badc](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

codecov-io commented on issue #7656: [AIRFLOW-7013] Automated check if Breeze 
image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#issuecomment-597323139
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=h1) 
Report
   > Merging 
[#7656](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/fcfc2aa7317916b402340e223b4431f87d9d6d6f=desc)
 will **decrease** coverage by `0.64%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7656/graphs/tree.svg?width=650=150=pr=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#7656  +/-   ##
   ==
   - Coverage   86.98%   86.34%   -0.65% 
   ==
 Files 904  904  
 Lines   4373643736  
   ==
   - Hits3804537764 -281 
   - Misses   5691 5972 +281 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=)
 | `21.51% <0.00%> (-72.16%)` | :arrow_down: |
   | 
[...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=)
 | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
   | 
[airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=)
 | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
   | 
[airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=)
 | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | 
[airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==)
 | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | 
[airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==)
 | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
   | 
[airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5)
 | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
   | 
[airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=)
 | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
   | 
[airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5)
 | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
   | ... and [10 
more](https://codecov.io/gh/apache/airflow/pull/7656/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=footer). 
Last update 
[fcfc2aa...115badc](https://codecov.io/gh/apache/airflow/pull/7656?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390602548
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -72,10 +75,11 @@ class RedshiftToS3Transfer(BaseOperator):
 @apply_defaults
 def __init__(  # pylint: disable=too-many-arguments
 self,
-schema: str,
-table: str,
 s3_bucket: str,
 s3_key: str,
+schema: str = None,
 
 Review comment:
   I see! I will document it in UPDATING.md 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar commented on a change in pull request #7688: [AIRFLOW-6794] Allow 
AWS Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390599830
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
 
 Review comment:
   Yes, I will work on adding the case, Thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow

potiuk commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger 
job from Airflow
URL: https://github.com/apache/airflow/pull/2708#issuecomment-597302478
 
 
   Looking forward to it! If you need any help with that - please let us know. 
Happy to help!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390591494
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -72,10 +75,11 @@ class RedshiftToS3Transfer(BaseOperator):
 @apply_defaults
 def __init__(  # pylint: disable=too-many-arguments
 self,
-schema: str,
-table: str,
 s3_bucket: str,
 s3_key: str,
+schema: str = None,
 
 Review comment:
   This is a breaking change  (someone could have used those as positional 
parameters) so it should be described in UPDATING.md. It's ok for 2.0 (we allow 
such backwards incompatibilities) but it must be described there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390591494
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -72,10 +75,11 @@ class RedshiftToS3Transfer(BaseOperator):
 @apply_defaults
 def __init__(  # pylint: disable=too-many-arguments
 self,
-schema: str,
-table: str,
 s3_bucket: str,
 s3_key: str,
+schema: str = None,
 
 Review comment:
   This is a breaking change so it should be described in UPDATING.md. It's ok 
for 2.0 (we allow such backwards incompatibilities) but it must be described 
there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

2020-03-10 Thread ASF subversion and git services (Jira)

potiuk commented on a change in pull request #7688: [AIRFLOW-6794] Allow AWS 
Operator RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688#discussion_r390593284
 
 

 ##
 File path: airflow/providers/amazon/aws/operators/redshift_to_s3.py
 ##
 @@ -107,7 +112,12 @@ def execute(self, context):
 credentials = s3_hook.get_credentials()
 unload_options = '\n\t\t\t'.join(self.unload_options)
 s3_key = '{}/{}_'.format(self.s3_key, self.table) if 
self.table_as_file_name else self.s3_key
-select_query = "SELECT * FROM 
{schema}.{table}".format(schema=self.schema, table=self.table)
+if self.custom_select_query is None:
 
 Review comment:
   We need  a test case covering that case. there is a test already in 
TestRedshiftToS3Transfer and you can add new one with custom select query 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7037) Fix Incorrect Type Annotation for Multiprocessing Connection



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056394#comment-17056394
 ] 

ASF subversion and git services commented on AIRFLOW-7037:
--

Commit 2536b517e6b8d9b54289beb7d0da4fee809e0fb5 in airflow's branch 
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2536b51 ]

[AIRFLOW-7037] Fix Incorrect Type Annotation for Multiprocessing Connection 
(#7687)



> Fix Incorrect Type Annotation for Multiprocessing Connection
> 
>
> Key: AIRFLOW-7037
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7037
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> The current annotation for *DagFileProcessorManager.signal_conn* is incorrect.
> Currently, it shows airflow.models.connection instead of 
> *multiprocessing.connection.Connection*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-7037) Fix Incorrect Type Annotation for Multiprocessing Connection

2020-03-10 Thread Jarek Potiuk (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk resolved AIRFLOW-7037.
---
Resolution: Fixed

> Fix Incorrect Type Annotation for Multiprocessing Connection
> 
>
> Key: AIRFLOW-7037
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7037
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> The current annotation for *DagFileProcessorManager.signal_conn* is incorrect.
> Currently, it shows airflow.models.connection instead of 
> *multiprocessing.connection.Connection*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-7037) Fix Incorrect Type Annotation for Multiprocessing Connection



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056393#comment-17056393
 ] 

ASF GitHub Bot commented on AIRFLOW-7037:
-

potiuk commented on pull request #7687: [AIRFLOW-7037] Fix Type Annotation for 
Multiprocessing Connection
URL: https://github.com/apache/airflow/pull/7687
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix Incorrect Type Annotation for Multiprocessing Connection
> 
>
> Key: AIRFLOW-7037
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7037
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> The current annotation for *DagFileProcessorManager.signal_conn* is incorrect.
> Currently, it shows airflow.models.connection instead of 
> *multiprocessing.connection.Connection*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk merged pull request #7687: [AIRFLOW-7037] Fix Type Annotation for Multiprocessing Connection

potiuk merged pull request #7687: [AIRFLOW-7037] Fix Type Annotation for 
Multiprocessing Connection
URL: https://github.com/apache/airflow/pull/7687
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7687: [AIRFLOW-7037] Fix Type Annotation for Multiprocessing Connection

codecov-io commented on issue #7687: [AIRFLOW-7037] Fix Type Annotation for 
Multiprocessing Connection
URL: https://github.com/apache/airflow/pull/7687#issuecomment-597293858
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=h1) 
Report
   > Merging 
[#7687](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e6af02fdf4fa431739f49e444c93c54ca9f5a8e0?src=pr=desc)
 will **decrease** coverage by `22.44%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7687/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#7687   +/-   ##
   ===
   - Coverage   86.99%   64.55%   -22.45% 
   ===
 Files 904  903-1 
 Lines   4372843723-5 
   ===
   - Hits3804328227 -9816 
   - Misses   568515496 +9811
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==)
 | `96.46% <100%> (+0.12%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `25.81% <100%> (-62.53%)` | :arrow_down: |
   | 
[...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [489 
more](https://codecov.io/gh/apache/airflow/pull/7687/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=footer). 
Last update 
[e6af02f...158d24c](https://codecov.io/gh/apache/airflow/pull/7687?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] blcksrx opened a new pull request #7689: [AIRFLOW-7038] Cassandra sensors tests

blcksrx opened a new pull request #7689: [AIRFLOW-7038] Cassandra sensors tests
URL: https://github.com/apache/airflow/pull/7689
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7038) Cassandra sensors tests



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056338#comment-17056338
 ] 

ASF GitHub Bot commented on AIRFLOW-7038:
-

blcksrx commented on pull request #7689: [AIRFLOW-7038] Cassandra sensors tests
URL: https://github.com/apache/airflow/pull/7689
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Cassandra sensors tests
> ---
>
> Key: AIRFLOW-7038
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7038
> Project: Apache Airflow
>  Issue Type: Test
>  Components: tests
>Affects Versions: 1.10.9
>Reporter: Sayed Mohammad Hossein Torabi
>Assignee: Sayed Mohammad Hossein Torabi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-7038) Cassandra sensors tests

2020-03-10 Thread Sayed Mohammad Hossein Torabi (Jira)

Sayed Mohammad Hossein Torabi created AIRFLOW-7038:
--

 Summary: Cassandra sensors tests
 Key: AIRFLOW-7038
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7038
 Project: Apache Airflow
  Issue Type: Test
  Components: tests
Affects Versions: 1.10.9
Reporter: Sayed Mohammad Hossein Torabi
Assignee: Sayed Mohammad Hossein Torabi






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] swiftomkar opened a new pull request #7688: [AIRFLOW-6794] Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

swiftomkar opened a new pull request #7688: [AIRFLOW-6794] Allow AWS Operator 
RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-6794) Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056328#comment-17056328
 ] 

ASF GitHub Bot commented on AIRFLOW-6794:
-

swiftomkar commented on pull request #7688: [AIRFLOW-6794] Allow AWS Operator 
RedshiftToS3Transfer To Run a Custom Query
URL: https://github.com/apache/airflow/pull/7688
 
 
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
> -
>
> Key: AIRFLOW-6794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6794
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, operators
>Affects Versions: 1.10.9
>Reporter: Roger Russel Droique Neris
>Assignee: Omkar
>Priority: Trivial
>  Labels: AWS, newbie
>
> {{The Redshift operator }}
> {{"airflow.providers.amazon.aws.operators.redshift_to_s3.}}{{RedshiftToS3Transfer"
>  allow only a simple usage to transfer a table to a S3. }}
> {{[https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/operators/redshift_to_s3.py#L110]}}
> {{If possible I would like to implement an usage of a custom query on it.}}
> {{The behavior expected is when a "query" parameter is given then it wil use 
> it, and if the it was not given it will use the default behavior.}}
> {{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (AIRFLOW-6794) Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query

2020-03-10 Thread Omkar (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Omkar reassigned AIRFLOW-6794:
--

Assignee: Omkar  (was: Roger Russel Droique Neris)

> Allow AWS Operator RedshiftToS3Transfer To Run a Custom Query
> -
>
> Key: AIRFLOW-6794
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6794
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, operators
>Affects Versions: 1.10.9
>Reporter: Roger Russel Droique Neris
>Assignee: Omkar
>Priority: Trivial
>  Labels: AWS, newbie
>
> {{The Redshift operator }}
> {{"airflow.providers.amazon.aws.operators.redshift_to_s3.}}{{RedshiftToS3Transfer"
>  allow only a simple usage to transfer a table to a S3. }}
> {{[https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/operators/redshift_to_s3.py#L110]}}
> {{If possible I would like to implement an usage of a custom query on it.}}
> {{The behavior expected is when a "query" parameter is given then it wil use 
> it, and if the it was not given it will use the default behavior.}}
> {{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

potiuk commented on a change in pull request #7656: [AIRFLOW-7013] Automated 
check if Breeze image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#discussion_r390568970
 
 

 ##
 File path: scripts/ci/_utils.sh
 ##
 @@ -552,6 +561,85 @@ function set_current_image_variables {
 fi
 }
 
+function build_image_manifest() {
+verbose_docker inspect "${AIRFLOW_IMAGE}" > 
"manifests/${AIRFLOW_BASE_TAG}.json"
+verbose_docker build \
+--build-arg AIRFLOW_BASE_TAG="${AIRFLOW_BASE_TAG}" \
+--tag="${AIRFLOW_LOCAL_MANIFEST_IMAGE}" \
+-f- . &1
+if ! verbose_docker create --name "local-airflow-manifest" \
+"${AIRFLOW_LOCAL_MANIFEST_IMAGE}"  >/dev/null 2>&1 ; then
+echo
+echo "Local docker not available"
 
 Review comment:
   Added full description + a bit nicer output showing pull progress as well as 
step of the docker build during pre-commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] jmcarp commented on issue #7667: [AIRFLOW-7017] Respect default dag view in trigger dag origin.

jmcarp commented on issue #7667: [AIRFLOW-7017] Respect default dag view in 
trigger dag origin.
URL: https://github.com/apache/airflow/pull/7667#issuecomment-597278290
 
 
   Thanks @kaxil, I added regression tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4175) S3Hook load_file should support ACL policy parameter

2020-03-10 Thread Ephraim E Anierobi (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056307#comment-17056307
 ] 

Ephraim E Anierobi commented on AIRFLOW-4175:
-

Hello [~jackjack10] , Can I submit a PR on this as part of my outreachy 
application?

> S3Hook load_file should support ACL policy parameter
> 
>
> Key: AIRFLOW-4175
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4175
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.10.2
>Reporter: Keith O'Brien
>Priority: Major
>  Labels: gsoc, gsoc2020, mentor
>
> We have a use case where we are uploading files to an S3 bucket in a 
> different AWS account to the one Airflow is running in.  AWS S3 supports this 
> situation using the pre canned ACL policy, specifically 
> {{bucket-owner-full-control. }}
> However, the current implementations of the {{S3Hook.load_*}}() and 
> {{S3Hook.copy_object}}() methods do not allow us to supply any ACL policy for 
> the file being uploaded/copied to S3.  
> It would be good to add another optional parameter to the {{S3Hook}} methods 
> called {{acl_policy}} which would then be passed into the boto3 client method 
> calls like so 
>  
> {code}
> # load_file
> ...
> if encrypt: 
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
>  
> {code}
> # load_bytes
> ...
> if encrypt: 
>extra_args['ServerSideEncryption'] = "AES256"
> if acl_policy:
>extra_args['ACL'] = acl_policy
> client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args){code}
> {code}
> # copy_object
> self.get_conn().copy_object(Bucket=dest_bucket_name,
>Key=dest_bucket_key,
>CopySource=CopySource, 
>ACL=acl_policy)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] codecov-io edited a comment on issue #7227: [AIRFLOW-6530] Allow Custom Statsd Client

codecov-io edited a comment on issue #7227: [AIRFLOW-6530] Allow Custom Statsd 
Client
URL: https://github.com/apache/airflow/pull/7227#issuecomment-577151472
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=h1) 
Report
   > Merging 
[#7227](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/2ea9278f76bf71aafb5601160602bf7f4194242f?src=pr=desc)
 will **decrease** coverage by `22.37%`.
   > The diff coverage is `71.16%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7227/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#7227   +/-   ##
   ===
   - Coverage   86.83%   64.45%   -22.38% 
   ===
 Files 897  903+6 
 Lines   4275143735  +984 
   ===
   - Hits3712128189 -8932 
   - Misses   563015546 +9916
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...ders/google/cloud/example\_dags/example\_dataflow.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFmbG93LnB5)
 | `0% <ø> (-100%)` | :arrow_down: |
   | 
[airflow/serialization/serialized\_objects.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9zZXJpYWxpemF0aW9uL3NlcmlhbGl6ZWRfb2JqZWN0cy5weQ==)
 | `81.08% <ø> (-9.13%)` | :arrow_down: |
   | 
[...\_platform/example\_dags/example\_campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9jYW1wYWlnbl9tYW5hZ2VyLnB5)
 | `93.93% <ø> (ø)` | :arrow_up: |
   | 
[...rflow/providers/amazon/aws/operators/sftp\_to\_s3.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2Z0cF90b19zMy5weQ==)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/providers/sftp/operators/sftp.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc2Z0cC9vcGVyYXRvcnMvc2Z0cC5weQ==)
 | `87.67% <ø> (ø)` | :arrow_up: |
   | 
[airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==)
 | `75.35% <ø> (-19.72%)` | :arrow_down: |
   | 
[...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==)
 | `93.63% <ø> (ø)` | :arrow_up: |
   | 
[airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==)
 | `85.46% <ø> (-0.06%)` | :arrow_down: |
   | 
[...cncf/kubernetes/example\_dags/example\_kubernetes.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2V4YW1wbGVfZGFncy9leGFtcGxlX2t1YmVybmV0ZXMucHk=)
 | `0% <0%> (-78.58%)` | :arrow_down: |
   | 
[...s/google/cloud/example\_dags/example\_datacatalog.py](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFjYXRhbG9nLnB5)
 | `0% <0%> (ø)` | |
   | ... and [538 
more](https://codecov.io/gh/apache/airflow/pull/7227/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=footer). 
Last update 
[2ea9278...b909492](https://codecov.io/gh/apache/airflow/pull/7227?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-7035) Increment the pip version in Airflow Dockerfile

2020-03-10 Thread Aditya (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya updated AIRFLOW-7035:

Description: 
The current pip version in 
([https://github.com/apache/airflow/blob/master/Dockerfile#L298)] does not 
support some  APIs in Airflow.

Eg: tableauhyperapi gets the following error while being installed in GCP 
Composer environemnt.

```

No matching distribution found for tableauhyperapi (from -r requirements.txt 
(line 3))"

```

This same error gets resolved when we locally upgrade pip to 20.0.2 .

Will it be possible to do the upgrade in the Airflow Docker File, so that 
Airflow can also support this (and other) APIs? 

 

  was:
The current pip version in 
([https://github.com/apache/airflow/blob/master/Dockerfile#L298)] does not 
support some  APIs in Airflow.

Eg: tableauhyperapi gets the following error while being installed in GCP 
Composer environemnt.

```

No matching distribution found for tableauhyperapi (from -r requirements.txt 
(line 3))"

```

This same error gets resolved when we locally upgrade pip to 20.0.2 .

Will it be possible to do the same in the Airflow Docker File, so that Airflow 
can also support this (and other) APIs? 

 


> Increment the pip version in Airflow Dockerfile
> ---
>
> Key: AIRFLOW-7035
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7035
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 1.10.9
>Reporter: Aditya
>Priority: Minor
>
> The current pip version in 
> ([https://github.com/apache/airflow/blob/master/Dockerfile#L298)] does not 
> support some  APIs in Airflow.
> Eg: tableauhyperapi gets the following error while being installed in GCP 
> Composer environemnt.
> ```
> No matching distribution found for tableauhyperapi (from -r requirements.txt 
> (line 3))"
> ```
> This same error gets resolved when we locally upgrade pip to 20.0.2 .
> Will it be possible to do the upgrade in the Airflow Docker File, so that 
> Airflow can also support this (and other) APIs? 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7687: [AIRFLOW-7037] Fix Type Annotation for Multiprocessing Connection

kaxil opened a new pull request #7687: [AIRFLOW-7037] Fix Type Annotation for 
Multiprocessing Connection
URL: https://github.com/apache/airflow/pull/7687
 
 
   The current annotation for DagFileProcessorManager.signal_conn is incorrect.
   
   Currently, it shows airflow.models.connection instead of 
multiprocessing.connection.Connection
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7037) Fix Incorrect Type Annotation for Multiprocessing Connection



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056271#comment-17056271
 ] 

ASF GitHub Bot commented on AIRFLOW-7037:
-

kaxil commented on pull request #7687: [AIRFLOW-7037] Fix Type Annotation for 
Multiprocessing Connection
URL: https://github.com/apache/airflow/pull/7687
 
 
   The current annotation for DagFileProcessorManager.signal_conn is incorrect.
   
   Currently, it shows airflow.models.connection instead of 
multiprocessing.connection.Connection
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix Incorrect Type Annotation for Multiprocessing Connection
> 
>
> Key: AIRFLOW-7037
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7037
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> The current annotation for *DagFileProcessorManager.signal_conn* is incorrect.
> Currently, it shows airflow.models.connection instead of 
> *multiprocessing.connection.Connection*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] yishan-lin commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow

yishan-lin commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to 
trigger job from Airflow
URL: https://github.com/apache/airflow/pull/2708#issuecomment-597244763
 
 
   This issue got closed by stalebot, but a first-class Airflow integration is 
on the roadmap for us internally.  We're working on some other integrations 
first (e.g Spinnaker) before we'll get there but still in plans!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-7037) Fix Incorrect Type Annotation for Multiprocessing Connection

2020-03-10 Thread ASF subversion and git services (Jira)

Kaxil Naik created AIRFLOW-7037:
---

 Summary: Fix Incorrect Type Annotation for Multiprocessing 
Connection
 Key: AIRFLOW-7037
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7037
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 2.0.0
Reporter: Kaxil Naik
Assignee: Kaxil Naik
 Fix For: 2.0.0


The current annotation for *DagFileProcessorManager.signal_conn* is incorrect.

Currently, it shows airflow.models.connection instead of 
*multiprocessing.connection.Connection*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-7036) redshift_copy_operator

2020-03-10 Thread Omkar (Jira)

Omkar created AIRFLOW-7036:
--

 Summary: redshift_copy_operator
 Key: AIRFLOW-7036
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7036
 Project: Apache Airflow
  Issue Type: Improvement
  Components: aws, operators
Affects Versions: 1.10.9
Reporter: Omkar
Assignee: Omkar


The s3_to_redshift operator in its current state is limited. I think we should 
add the option of uploading data from dynamoDB to redshift as well. I would 
also like to consider a way to do upserts into redshift. Would be happy to 
discuss this in more detail with folks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-7035) Increment the pip version in Airflow Dockerfile

2020-03-10 Thread Aditya (Jira)

Aditya created AIRFLOW-7035:
---

 Summary: Increment the pip version in Airflow Dockerfile
 Key: AIRFLOW-7035
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7035
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 1.10.9
Reporter: Aditya


The current pip version in 
([https://github.com/apache/airflow/blob/master/Dockerfile#L298)] does not 
support some  APIs in Airflow.

Eg: tableauhyperapi gets the following error while being installed in GCP 
Composer environemnt.

```

No matching distribution found for tableauhyperapi (from -r requirements.txt 
(line 3))"

```

This same error gets resolved when we locally upgrade pip to 20.0.2 .

Will it be possible to do the same in the Airflow Docker File, so that Airflow 
can also support this (and other) APIs? 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6809) Test for presto operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056220#comment-17056220
 ] 

ASF subversion and git services commented on AIRFLOW-6809:
--

Commit fcfc2aa7317916b402340e223b4431f87d9d6d6f in airflow's branch 
refs/heads/master from Hossein Torabi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fcfc2aa ]

 [AIRFLOW-6809] Add tests for presto operators (#7422)



> Test for presto operators
> -
>
> Key: AIRFLOW-6809
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6809
> Project: Apache Airflow
>  Issue Type: Test
>  Components: tests
>Affects Versions: 1.10.9
>Reporter: Sayed Mohammad Hossein Torabi
>Assignee: Sayed Mohammad Hossein Torabi
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-6809) Test for presto operators



[ 
https://issues.apache.org/jira/browse/AIRFLOW-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056219#comment-17056219
 ] 

ASF GitHub Bot commented on AIRFLOW-6809:
-

kaxil commented on pull request #7422: [AIRFLOW-6809] Test for presto operators
URL: https://github.com/apache/airflow/pull/7422
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Test for presto operators
> -
>
> Key: AIRFLOW-6809
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6809
> Project: Apache Airflow
>  Issue Type: Test
>  Components: tests
>Affects Versions: 1.10.9
>Reporter: Sayed Mohammad Hossein Torabi
>Assignee: Sayed Mohammad Hossein Torabi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AIRFLOW-6809) Test for presto operators



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-6809.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Test for presto operators
> -
>
> Key: AIRFLOW-6809
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6809
> Project: Apache Airflow
>  Issue Type: Test
>  Components: tests
>Affects Versions: 1.10.9
>Reporter: Sayed Mohammad Hossein Torabi
>Assignee: Sayed Mohammad Hossein Torabi
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil merged pull request #7422: [AIRFLOW-6809] Test for presto operators

kaxil merged pull request #7422: [AIRFLOW-6809] Test for presto operators
URL: https://github.com/apache/airflow/pull/7422
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7686: New user company

2020-03-10 Thread ASF subversion and git services (Jira)

boring-cyborg[bot] commented on issue #7686: New user company
URL: https://github.com/apache/airflow/pull/7686#issuecomment-597235073

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for
testing locally, it’s a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
- Be sure to read the [Airflow Coding style](
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7032) Remove airflow/gcp folder from doc build



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056208#comment-17056208
 ] 

ASF subversion and git services commented on AIRFLOW-7032:
--

Commit 2440c4a54b45e61b3f50fb10ac6b266d9785f142 in airflow's branch 
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2440c4a ]

[AIRFLOW-7032] Remove airflow/gcp folder from doc build (#7681)



> Remove airflow/gcp folder from doc build
> 
>
> Key: AIRFLOW-7032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7032
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> Currently, our docs build have the following warning/error:
> {code}
> find: ‘../airflow/gcp/operators’: No such file or directory
> find: ‘../airflow/gcp/sensors’: No such file or directory
> find: ‘../airflow/gcp/hooks’: No such file or directory
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] Better-Boy opened a new pull request #7686: New user company

Better-Boy opened a new pull request #7686: New user company
URL: https://github.com/apache/airflow/pull/7686
 
 
   Added company that heavily uses airflow
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-7032) Remove airflow/gcp folder from doc build



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik resolved AIRFLOW-7032.
-
Fix Version/s: 2.0.0
   Resolution: Fixed

> Remove airflow/gcp folder from doc build
> 
>
> Key: AIRFLOW-7032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7032
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> Currently, our docs build have the following warning/error:
> {code}
> find: ‘../airflow/gcp/operators’: No such file or directory
> find: ‘../airflow/gcp/sensors’: No such file or directory
> find: ‘../airflow/gcp/hooks’: No such file or directory
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil merged pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder from doc build

kaxil merged pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder from 
doc build
URL: https://github.com/apache/airflow/pull/7681
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7032) Remove airflow/gcp folder from doc build



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056206#comment-17056206
 ] 

ASF GitHub Bot commented on AIRFLOW-7032:
-

kaxil commented on pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder 
from doc build
URL: https://github.com/apache/airflow/pull/7681
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove airflow/gcp folder from doc build
> 
>
> Key: AIRFLOW-7032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7032
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
>
> Currently, our docs build have the following warning/error:
> {code}
> find: ‘../airflow/gcp/operators’: No such file or directory
> find: ‘../airflow/gcp/sensors’: No such file or directory
> find: ‘../airflow/gcp/hooks’: No such file or directory
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] OmairK commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

OmairK commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680#issuecomment-597227747
 
 
   Your welcome @potiuk  :smile: . Thanks a lot for guiding me through my first 
PR. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check

potiuk commented on issue #7678: [AIRFLOW-7029] Use separate docker image for 
running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597226479
 
 
   > @potiuk I don't see what rebasing on #7656 would do? Upon first look I 
thought it wouldn't matter which of the two are merged first...?
   
   Actually you are right even better to merge it first!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4438) Add Gzip compression to S3_hook

2020-03-10 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056189#comment-17056189
 ] 

ASF GitHub Bot commented on AIRFLOW-4438:
-

potiuk commented on pull request #7680: [AIRFLOW-4438] Add Gzip compression to 
S3_hook
URL: https://github.com/apache/airflow/pull/7680
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add Gzip compression to S3_hook
> ---
>
> Key: AIRFLOW-4438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4438
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, hooks
>Affects Versions: 1.10.3
>Reporter: jack
>Priority: Major
>  Labels: gsoc, gsoc2020, mentor
>
> Allow to load compressed file in the load_file function.
> We have similar logic in GoogleCloudStorageHook



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-4438) Add Gzip compression to S3_hook



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056190#comment-17056190
 ] 

ASF subversion and git services commented on AIRFLOW-4438:
--

Commit b7cdda1c64595bc7f85519337029de259e573fce in airflow's branch 
refs/heads/master from Omair Khan
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b7cdda1 ]

[AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)



> Add Gzip compression to S3_hook
> ---
>
> Key: AIRFLOW-4438
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4438
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, hooks
>Affects Versions: 1.10.3
>Reporter: jack
>Priority: Major
>  Labels: gsoc, gsoc2020, mentor
>
> Allow to load compressed file in the load_file function.
> We have similar logic in GoogleCloudStorageHook



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

boring-cyborg[bot] commented on issue #7680: [AIRFLOW-4438] Add Gzip 
compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680#issuecomment-597225024
 
 
   Awesome work, congrats on your first merged pull request!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

potiuk commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680#issuecomment-597225116
 
 
   Thanks @OmairK !
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

potiuk merged pull request #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

codecov-io commented on issue #7680: [AIRFLOW-4438] Add Gzip compression to 
S3_hook
URL: https://github.com/apache/airflow/pull/7680#issuecomment-597208124
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=h1) 
Report
   > Merging 
[#7680](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/e6af02fdf4fa431739f49e444c93c54ca9f5a8e0?src=pr=desc)
 will **decrease** coverage by `22.52%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/7680/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#7680   +/-   ##
   ===
   - Coverage   86.99%   64.47%   -22.53% 
   ===
 Files 904  903-1 
 Lines   4372843723-5 
   ===
   - Hits3804328189 -9854 
   - Misses   568515534 +9849
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/providers/amazon/aws/hooks/s3.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9zMy5weQ==)
 | `96.46% <100%> (+0.12%)` | :arrow_up: |
   | 
[...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...viders/docker/example\_dags/example\_docker\_swarm.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlcl9zd2FybS5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[...providers/google/cloud/example\_dags/example\_dlp.py](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RscC5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [487 
more](https://codecov.io/gh/apache/airflow/pull/7680/diff?src=pr=tree-more) 
| |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=footer). 
Last update 
[e6af02f...89b6207](https://codecov.io/gh/apache/airflow/pull/7680?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ushasree-tella closed pull request #7684: Update Breeze.rst

ushasree-tella closed pull request #7684: Update Breeze.rst
URL: https://github.com/apache/airflow/pull/7684
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7034) Remove feature: Assigning Dag to task using Bitshift Operator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056143#comment-17056143
 ] 

ASF GitHub Bot commented on AIRFLOW-7034:
-

kaxil commented on pull request #7685: [AIRFLOW-7034] Remove feature: Assigning 
Dag to task using Bitshift Op
URL: https://github.com/apache/airflow/pull/7685
 
 
   It is possible to assign a task to the dag using the bitshift operators, 
however it doesn't pick up default_args when done this way 
:
   
   ```
   dag = DAG('my_dag', default_args=default_args)
   dummy = DummyOperator(task_id='dummy')
   
   dag >> dummy
   ```
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove feature: Assigning Dag to task using Bitshift Operator
> -
>
> Key: AIRFLOW-7034
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7034
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> It is possible to assign a task to the dag using the bitshift operators, 
> however it doesn't pick up default_args when done this way 
> :
> ```
> dag = DAG('my_dag', default_args=default_args)
> dummy = DummyOperator(task_id='dummy')
> dag >> dummy
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7685: [AIRFLOW-7034] Remove feature: Assigning Dag to task using Bitshift Op

kaxil opened a new pull request #7685: [AIRFLOW-7034] Remove feature: Assigning 
Dag to task using Bitshift Op
URL: https://github.com/apache/airflow/pull/7685
 
 
   It is possible to assign a task to the dag using the bitshift operators, 
however it doesn't pick up default_args when done this way 
:
   
   ```
   dag = DAG('my_dag', default_args=default_args)
   dummy = DummyOperator(task_id='dummy')
   
   dag >> dummy
   ```
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-7034) Remove feature: Assigning Dag to task using Bitshift Operator

Kaxil Naik created AIRFLOW-7034:
---

 Summary: Remove feature: Assigning Dag to task using Bitshift 
Operator
 Key: AIRFLOW-7034
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7034
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: 2.0.0
Reporter: Kaxil Naik
Assignee: Kaxil Naik
 Fix For: 2.0.0


It is possible to assign a task to the dag using the bitshift operators, 
however it doesn't pick up default_args when done this way 
:

```
dag = DAG('my_dag', default_args=default_args)
dummy = DummyOperator(task_id='dummy')

dag >> dummy
```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ushasree-tella opened a new pull request #7684: Update Breeze.rst

ushasree-tella opened a new pull request #7684: Update Breeze.rst
URL: https://github.com/apache/airflow/pull/7684
 
 
   Updated command to create user prior to running the webserver in order to 
log in
   ---
   Issue link: AIRFLOW-
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In the case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7684: Update Breeze.rst

boring-cyborg[bot] commented on issue #7684: Update Breeze.rst
URL: https://github.com/apache/airflow/pull/7684#issuecomment-597202837

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Work stopped] (AIRFLOW-7028) Remove java from inside CI image

2020-03-10 Thread Ash Berlin-Taylor (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-7028 stopped by Ash Berlin-Taylor.
--
> Remove java from inside CI image
> 
>
> Key: AIRFLOW-7028
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7028
> Project: Apache Airflow
>  Issue Type: Task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
> A noticable portion of the time spend running static checks is pulling the 
> docker image down (it takes about 3 of the 7-8 minutes!) so if we can reduce 
> the size of the image our tests will be quicker.
> Almost 1/3 of the size of the image is from Java, which is not needed for 
> anything core to Airflow, and the bits that do need it should be converted to 
> system test etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (AIRFLOW-7028) Remove java from inside CI image

2020-03-10 Thread Ash Berlin-Taylor (Jira)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-7028 started by Ash Berlin-Taylor.
--
> Remove java from inside CI image
> 
>
> Key: AIRFLOW-7028
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7028
> Project: Apache Airflow
>  Issue Type: Task
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Minor
>
> A noticable portion of the time spend running static checks is pulling the 
> docker image down (it takes about 3 of the 7-8 minutes!) so if we can reduce 
> the size of the image our tests will be quicker.
> Almost 1/3 of the size of the image is from Java, which is not needed for 
> anything core to Airflow, and the bits that do need it should be converted to 
> system test etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on a change in pull request #7656: [AIRFLOW-7013] Automated check if Breeze image needs to be pulled

potiuk commented on a change in pull request #7656: [AIRFLOW-7013] Automated 
check if Breeze image needs to be pulled
URL: https://github.com/apache/airflow/pull/7656#discussion_r390460517
 
 

 ##
 File path: scripts/ci/_utils.sh
 ##
 @@ -552,6 +561,85 @@ function set_current_image_variables {
 fi
 }
 
+function build_image_manifest() {
+verbose_docker inspect "${AIRFLOW_IMAGE}" > 
"manifests/${AIRFLOW_BASE_TAG}.json"
+verbose_docker build \
+--build-arg AIRFLOW_BASE_TAG="${AIRFLOW_BASE_TAG}" \
+--tag="${AIRFLOW_LOCAL_MANIFEST_IMAGE}" \
+-f- . &1
+if ! verbose_docker create --name "local-airflow-manifest" \
+"${AIRFLOW_LOCAL_MANIFEST_IMAGE}"  >/dev/null 2>&1 ; then
+echo
+echo "Local docker not available"
 
 Review comment:
   > Can you add some docs/comments in the PR somewhere about
   > 
   > 1. what this is doing -- I'm having trouble following the logic; and
   > 2. why (your comment in the PR should go in the script somewhere)
   
   Good point. I added comments describing both.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7678: [AIRFLOW-7029] Use separate docker image for running license check

ashb commented on issue #7678: [AIRFLOW-7029] Use separate docker image for 
running license check
URL: https://github.com/apache/airflow/pull/7678#issuecomment-597192834
 
 
   @potiuk I don't see what rebasing on #7656 would do? Upon first look I 
thought it wouldn't matter which of the two are merged first...?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7033) Change dag and task state meanwhile



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056118#comment-17056118
 ] 

ASF GitHub Bot commented on AIRFLOW-7033:
-

zhongjiajie commented on pull request #7683: [AIRFLOW-7033] Change dag and task 
state meanwhile
URL: https://github.com/apache/airflow/pull/7683
 
 
   In dag page, have make task instance state to success
   or failed bottom, when we click those bottom not only
   change task instance state but also dag run state
   if possible.
   
   * When trigger make_failed bottom, the dag run state
 should change to failed
   * When trigger make_success bottom, then dag run
 state should change to success if all task instance
 state are success, otherwise should keep the
 original state
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Change dag and task state meanwhile
> ---
>
> Key: AIRFLOW-7033
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7033
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: api, webserver
>Affects Versions: 1.10.9
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>
> In dag page, have make task instance state to success or failed bottom, when 
> we click those bottom not only also change task instance state but also dag 
> run state if possible.
>  * When trigger make_failed bottom, the dag run state should change to failed
>  * When trigger make_success bottom, then dag run state should change to 
> success if all task instance state are success, otherwise should keep the 
> original state



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] zhongjiajie opened a new pull request #7683: [AIRFLOW-7033] Change dag and task state meanwhile

zhongjiajie opened a new pull request #7683: [AIRFLOW-7033] Change dag and task 
state meanwhile
URL: https://github.com/apache/airflow/pull/7683
 
 
   In dag page, have make task instance state to success
   or failed bottom, when we click those bottom not only
   change task instance state but also dag run state
   if possible.
   
   * When trigger make_failed bottom, the dag run state
 should change to failed
   * When trigger make_success bottom, then dag run
 state should change to success if all task instance
 state are success, otherwise should keep the
 original state
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-7033) Change dag and task state meanwhile

2020-03-10 Thread zhongjiajie (Jira)

zhongjiajie created AIRFLOW-7033:


 Summary: Change dag and task state meanwhile
 Key: AIRFLOW-7033
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7033
 Project: Apache Airflow
  Issue Type: Improvement
  Components: api, webserver
Affects Versions: 1.10.9
Reporter: zhongjiajie
Assignee: zhongjiajie


In dag page, have make task instance state to success or failed bottom, when we 
click those bottom not only also change task instance state but also dag run 
state if possible.
 * When trigger make_failed bottom, the dag run state should change to failed
 * When trigger make_success bottom, then dag run state should change to 
success if all task instance state are success, otherwise should keep the 
original state



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] mik-laj commented on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file

mik-laj commented on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be 
configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#issuecomment-597176715
 
 
   We are not planning to add new features to Airflow 1.10.x, because Airflow 
2.0 and Airflow 1.10.x are very different, so cherry-picking changes is very 
problematic. In Airflow 1.10.x  At Airflow 1.10.x, we focus only on fixing 
existing bugs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Issue Comment Deleted] (AIRFLOW-7031) Airflow WinRM endpoint is hardcoded to HTTP



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noël BARDELOT updated AIRFLOW-7031:
---
Comment: was deleted

(was: See PR : https://github.com/apache/airflow/pull/7682)

> Airflow WinRM endpoint is hardcoded to HTTP
> ---
>
> Key: AIRFLOW-7031
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7031
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
> hardcoded to use 'http' even when the service is set to use 'https':
>  
> {{ # If endpoint is not set, then build a standard wsman endpoint from host 
> and port.}}
> {{ if not self.endpoint:}}
> {{   self.endpoint = 'http://\{0}:\{1}/wsman'.format(self.remote_host, 
> self.remote_port)}}
>  
> Workaround: configure the 'enpoint' setting manually in the extra 
> configuration of the connection.
> Correction: make the protocol configurable, depending on the value of the 
> 'service' configuration (if 'HTTP' then 'http://' should be used, and if 
> 'HTTPS' then 'https://' should be used).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-7031) Airflow WinRM endpoint is hardcoded to HTTP



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056101#comment-17056101
 ] 

Noël BARDELOT commented on AIRFLOW-7031:


See PR : https://github.com/apache/airflow/pull/7682

> Airflow WinRM endpoint is hardcoded to HTTP
> ---
>
> Key: AIRFLOW-7031
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7031
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
> hardcoded to use 'http' even when the service is set to use 'https':
>  
> {{ # If endpoint is not set, then build a standard wsman endpoint from host 
> and port.}}
> {{ if not self.endpoint:}}
> {{   self.endpoint = 'http://\{0}:\{1}/wsman'.format(self.remote_host, 
> self.remote_port)}}
>  
> Workaround: configure the 'enpoint' setting manually in the extra 
> configuration of the connection.
> Correction: make the protocol configurable, depending on the value of the 
> 'service' configuration (if 'HTTP' then 'http://' should be used, and if 
> 'HTTPS' then 'https://' should be used).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-7031) Airflow WinRM endpoint is hardcoded to HTTP



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056096#comment-17056096
 ] 

ASF GitHub Bot commented on AIRFLOW-7031:
-

NBardelot commented on pull request #7682: [AIRFLOW-7031] Airflow WinRM 
endpoint is hardcoded to HTTP
URL: https://github.com/apache/airflow/pull/7682
 
 
This commit fixes the issue and avoid errors like this:
   
   ```
   ERROR - Error connecting to host: 1.2.3.4, error: 
HTTPConnectionPool(host='1.2.3.4', port=5986): Read timed out. (read timeout=30)
   ```
   
   Even when `service` is set to `HTTPS` because the WinRM hook `endpoint`'s 
default configuration was hardcoded with the `http://` protocol.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Airflow WinRM endpoint is hardcoded to HTTP
> ---
>
> Key: AIRFLOW-7031
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7031
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
> hardcoded to use 'http' even when the service is set to use 'https':
>  
> {{ # If endpoint is not set, then build a standard wsman endpoint from host 
> and port.}}
> {{ if not self.endpoint:}}
> {{   self.endpoint = 'http://\{0}:\{1}/wsman'.format(self.remote_host, 
> self.remote_port)}}
>  
> Workaround: configure the 'enpoint' setting manually in the extra 
> configuration of the connection.
> Correction: make the protocol configurable, depending on the value of the 
> 'service' configuration (if 'HTTP' then 'http://' should be used, and if 
> 'HTTPS' then 'https://' should be used).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] NBardelot opened a new pull request #7682: [AIRFLOW-7031] Airflow WinRM endpoint is hardcoded to HTTP

NBardelot opened a new pull request #7682: [AIRFLOW-7031] Airflow WinRM 
endpoint is hardcoded to HTTP
URL: https://github.com/apache/airflow/pull/7682
 
 
This commit fixes the issue and avoid errors like this:
   
   ```
   ERROR - Error connecting to host: 1.2.3.4, error: 
HTTPConnectionPool(host='1.2.3.4', port=5986): Read timed out. (read timeout=30)
   ```
   
   Even when `service` is set to `HTTPS` because the WinRM hook `endpoint`'s 
default configuration was hardcoded with the `http://` protocol.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7674: [AIRFLOW-7022] Simplify DagFileProcessor.process_file method

mik-laj commented on issue #7674: [AIRFLOW-7022] Simplify 
DagFileProcessor.process_file method
URL: https://github.com/apache/airflow/pull/7674#issuecomment-597174623
 
 
   Travis is green again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-7032) Remove airflow/gcp folder from doc build



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056083#comment-17056083
 ] 

ASF GitHub Bot commented on AIRFLOW-7032:
-

kaxil commented on pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder 
from doc build
URL: https://github.com/apache/airflow/pull/7681
 
 
   Currently, our docs build have the following warning/error:
   
   ```
   find: ‘../airflow/gcp/operators’: No such file or directory
   find: ‘../airflow/gcp/sensors’: No such file or directory
   find: ‘../airflow/gc
   ```
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove airflow/gcp folder from doc build
> 
>
> Key: AIRFLOW-7032
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7032
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ci
>Affects Versions: 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
>
> Currently, our docs build have the following warning/error:
> {code}
> find: ‘../airflow/gcp/operators’: No such file or directory
> find: ‘../airflow/gcp/sensors’: No such file or directory
> find: ‘../airflow/gcp/hooks’: No such file or directory
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] kaxil opened a new pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder from doc build

kaxil opened a new pull request #7681: [AIRFLOW-7032] Remove airflow/gcp folder 
from doc build
URL: https://github.com/apache/airflow/pull/7681
 
 
   Currently, our docs build have the following warning/error:
   
   ```
   find: ‘../airflow/gcp/operators’: No such file or directory
   find: ‘../airflow/gcp/sensors’: No such file or directory
   find: ‘../airflow/gc
   ```
   
   ---
   Issue link: WILL BE INSERTED BY 
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = 
JIRA ID*
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   * For document-only changes commit message can start with 
`[AIRFLOW-]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] OmairK commented on a change in pull request #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

OmairK commented on a change in pull request #7680: [AIRFLOW-4438] Add Gzip 
compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680#discussion_r390410635
 
 

 ##
 File path: airflow/providers/amazon/aws/hooks/s3.py
 ##
 @@ -442,6 +445,9 @@ def load_file(self,
 :param encrypt: If True, the file will be encrypted on the server-side
 by S3 and will be stored in an encrypted form while at rest in S3.
 :type encrypt: bool
+:param gzip: If True, the file will be compressed on the server-side
 
 Review comment:
   Got it, I will resolve this


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] halilduygulu commented on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file

halilduygulu commented on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to 
be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#issuecomment-597148805
 
 
   Is this airflow 2.0 only? will it be available in 1.x.x relases?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7680: [AIRFLOW-4438] Add Gzip compression to S3_hook

potiuk commented on a change in pull request #7680: [AIRFLOW-4438] Add Gzip 
compression to S3_hook
URL: https://github.com/apache/airflow/pull/7680#discussion_r390396616
 
 

 ##
 File path: airflow/providers/amazon/aws/hooks/s3.py
 ##
 @@ -442,6 +445,9 @@ def load_file(self,
 :param encrypt: If True, the file will be encrypted on the server-side
 by S3 and will be stored in an encrypted form while at rest in S3.
 :type encrypt: bool
+:param gzip: If True, the file will be compressed on the server-side
 
 Review comment:
   ```suggestion
   :param gzip: If True, the file will be compressed locally
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-7032) Remove airflow/gcp folder from doc build

Kaxil Naik created AIRFLOW-7032:
---

 Summary: Remove airflow/gcp folder from doc build
 Key: AIRFLOW-7032
 URL: https://issues.apache.org/jira/browse/AIRFLOW-7032
 Project: Apache Airflow
  Issue Type: Improvement
  Components: ci
Affects Versions: 2.0.0
Reporter: Kaxil Naik
Assignee: Kaxil Naik


Currently, our docs build have the following warning/error:


{code}
find: ‘../airflow/gcp/operators’: No such file or directory
find: ‘../airflow/gcp/sensors’: No such file or directory
find: ‘../airflow/gcp/hooks’: No such file or directory
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (AIRFLOW-7031) Airflow WinRM endpoint is hardcoded to HTTP



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noël BARDELOT updated AIRFLOW-7031:
---
Description: 
In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
hardcoded to use 'http' even when the service is set to use 'https':

 

{{ # If endpoint is not set, then build a standard wsman endpoint from host and 
port.}}
{{ if not self.endpoint:}}
{{   self.endpoint = 'http://\{0}:\{1}/wsman'.format(self.remote_host, 
self.remote_port)}}

 

Workaround: configure the 'enpoint' setting manually in the extra configuration 
of the connection.

Correction: make the protocol configurable, depending on the value of the 
'service' configuration (if 'HTTP' then 'http://' should be used, and if 
'HTTPS' then 'https://' should be used).

  was:
In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
hardcoded to use 'http' even when the service is set to use 'https':

# If endpoint is not set, then build a standard wsman endpoint from 
host and port.
if not self.endpoint:
self.endpoint = 'http://{0}:{1}/wsman'.format(self.remote_host, 
self.remote_port)

Workaround: configure the 'enpoint' setting manually in the extra configuration 
of the connection.

Correction: make the protocol configurable, depending on the value of the 
'service' configuration (if 'HTTP' then 'http://' should be used, and if 
'HTTPS' then 'https://' should be used).


> Airflow WinRM endpoint is hardcoded to HTTP
> ---
>
> Key: AIRFLOW-7031
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7031
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 1.10.9
>Reporter: Noël BARDELOT
>Priority: Minor
>
> In airflow/providers/microsoft/winrm/hooks/winrm.py the following code is 
> hardcoded to use 'http' even when the service is set to use 'https':
>  
> {{ # If endpoint is not set, then build a standard wsman endpoint from host 
> and port.}}
> {{ if not self.endpoint:}}
> {{   self.endpoint = 'http://\{0}:\{1}/wsman'.format(self.remote_host, 
> self.remote_port)}}
>  
> Workaround: configure the 'enpoint' setting manually in the extra 
> configuration of the connection.
> Correction: make the protocol configurable, depending on the value of the 
> 'service' configuration (if 'HTTP' then 'http://' should be used, and if 
> 'HTTPS' then 'https://' should be used).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AIRFLOW-7031) Airflow WinRM endpoint is hardcoded to HTTP