date:20190322

feng-tao commented on issue #4640: [RFC] [WIP] Feature/flask restful
URL: https://github.com/apache/airflow/pull/4640#issuecomment-475839577
 
 
   @drewsonne @ashb I think the AIP for this change has pass, what is the next 
step?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

feng-tao commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268382201
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -2300,9 +2297,35 @@ def _per_task_process(task, key, ti, session=None):
 self.log.debug('Adding %s to not_ready', ti)
 ti_status.not_ready.add(key)
 
-for task in self.dag.topological_sort():
-for key, ti in list(ti_status.to_run.items()):
-_per_task_process(task, key, ti)
+non_pool_slots = conf.getint('core', 
'non_pooled_backfill_task_slot_count')
+
+try:
+for task in self.dag.topological_sort():
+for key, ti in list(ti_status.to_run.items()):
+if task.task_id != ti.task_id:
+continue
+if task.pool:
+pool = session.query(models.Pool) \
+.filter(models.Pool.pool == task.pool) \
+.first()
+if not pool:
+raise PoolNotFound('Unknown pool: 
{}'.format(task.pool))
+
+open_slots = pool.open_slots(session=session)
+if open_slots <= 0:
+raise NoAvailablePoolSlot(
 
 Review comment:
   thanks. lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268381262
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -2300,9 +2297,35 @@ def _per_task_process(task, key, ti, session=None):
 self.log.debug('Adding %s to not_ready', ti)
 ti_status.not_ready.add(key)
 
-for task in self.dag.topological_sort():
-for key, ti in list(ti_status.to_run.items()):
-_per_task_process(task, key, ti)
+non_pool_slots = conf.getint('core', 
'non_pooled_backfill_task_slot_count')
+
+try:
+for task in self.dag.topological_sort():
+for key, ti in list(ti_status.to_run.items()):
+if task.task_id != ti.task_id:
+continue
+if task.pool:
+pool = session.query(models.Pool) \
+.filter(models.Pool.pool == task.pool) \
+.first()
+if not pool:
+raise PoolNotFound('Unknown pool: 
{}'.format(task.pool))
+
+open_slots = pool.open_slots(session=session)
+if open_slots <= 0:
+raise NoAvailablePoolSlot(
 
 Review comment:
   Whenever there is no available slots, backfill would sleep and retry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

feng-tao commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268381021
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -2300,9 +2297,35 @@ def _per_task_process(task, key, ti, session=None):
 self.log.debug('Adding %s to not_ready', ti)
 ti_status.not_ready.add(key)
 
-for task in self.dag.topological_sort():
-for key, ti in list(ti_status.to_run.items()):
-_per_task_process(task, key, ti)
+non_pool_slots = conf.getint('core', 
'non_pooled_backfill_task_slot_count')
+
+try:
+for task in self.dag.topological_sort():
+for key, ti in list(ti_status.to_run.items()):
+if task.task_id != ti.task_id:
+continue
+if task.pool:
+pool = session.query(models.Pool) \
+.filter(models.Pool.pool == task.pool) \
+.first()
+if not pool:
+raise PoolNotFound('Unknown pool: 
{}'.format(task.pool))
+
+open_slots = pool.open_slots(session=session)
+if open_slots <= 0:
+raise NoAvailablePoolSlot(
 
 Review comment:
   so now if there is no enough pool, the backfill will fail or retry?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4887: [AIRFLOW-4055] Add AWS SQS Sensor

zhongjiajie commented on a change in pull request #4887: [AIRFLOW-4055] Add AWS 
SQS Sensor
URL: https://github.com/apache/airflow/pull/4887#discussion_r268380566
 
 

 ##
 File path: airflow/contrib/sensors/aws_sqs_sensor.py
 ##
 @@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.contrib.hooks.aws_sqs_hook import SQSHook
+from airflow.exceptions import AirflowException
+
+
+class SQSSensor(BaseSensorOperator):
+"""
+Get messages from an SQS queue and passes them through XCom.
+
+:param aws_conn_id: AWS connection id
+:type aws_conn_id: str
+:param sqs_queue: The SQS queue
+:type sqs_queue: str
+:param max_messages: The maximum number of messages to retrieve for each 
poke
+:type max_messages: int
+"""
+
+template_fields = ['sqs_queue', 'max_messages']
 
 Review comment:
   ```suggestion
   template_fields = ('sqs_queue', 'max_messages')
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4887: [AIRFLOW-4055] Add AWS SQS Sensor

zhongjiajie commented on a change in pull request #4887: [AIRFLOW-4055] Add AWS 
SQS Sensor
URL: https://github.com/apache/airflow/pull/4887#discussion_r268380672
 
 

 ##
 File path: airflow/contrib/sensors/aws_sqs_sensor.py
 ##
 @@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.contrib.hooks.aws_sqs_hook import SQSHook
+from airflow.exceptions import AirflowException
+
+
+class SQSSensor(BaseSensorOperator):
+"""
+Get messages from an SQS queue and passes them through XCom.
+
+:param aws_conn_id: AWS connection id
+:type aws_conn_id: str
+:param sqs_queue: The SQS queue
+:type sqs_queue: str
+:param max_messages: The maximum number of messages to retrieve for each 
poke
+:type max_messages: int
+"""
+
+template_fields = ['sqs_queue', 'max_messages']
+
+@apply_defaults
+def __init__(self, sqs_queue, aws_conn_id='aws_default', max_messages=5, 
*args, **kwargs):
+super(SQSSensor, self).__init__(*args, **kwargs)
+self.sqs_queue = sqs_queue
+self.aws_conn_id = aws_conn_id
+self.max_messages = max_messages
+
+def poke(self, context):
+"""
+Check for message on subscribed queue and write to xcom the message 
with key ``messages``
+
+:param context: the context object
+:type context: dict
+:return: ``True`` if message is available or ``False``
+"""
+
+sqs_hook = SQSHook(aws_conn_id=self.aws_conn_id)
+self.log.info('SQSSensor checking for message on queue: %s', 
self.sqs_queue)
+
+try:
+messages = 
sqs_hook.get_conn().receive_message(QueueUrl=self.sqs_queue,
+   
MaxNumberOfMessages=self.max_messages)
+
+self.log.info("reveived message %s", str(messages))
+
+if 'Messages' not in messages:
+self.log.info('No message received %s', str(messages))
+return False
+
+if (len(messages['Messages']) > 0):
+context['ti'].xcom_push(key='messages', value=messages)
+
+entries = [{'Id': message['MessageId'], 'ReceiptHandle': 
message['ReceiptHandle']}
+   for message in messages['Messages']]
+
+result = 
sqs_hook.get_conn().delete_message_batch(QueueUrl=self.sqs_queue,
+  
Entries=entries)
+
+if ('Successful' in result):
 
 Review comment:
   ```suggestion
   if 'Successful' in result:
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #4829: [AIRFLOW-3993] Change SalesforceHook and add tests

feng-tao commented on issue #4829: [AIRFLOW-3993] Change SalesforceHook and add 
tests
URL: https://github.com/apache/airflow/pull/4829#issuecomment-475835675
 
 
   @feluelle , could you rebase your pr with master?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view dag details

feng-tao commented on issue #4569: [AIRFLOW-3745] Fix viewer not able to view 
dag details
URL: https://github.com/apache/airflow/pull/4569#issuecomment-475835209
 
 
   @ashb , pr posted (https://github.com/apache/airflow/pull/4960)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's

feng-tao commented on issue #4808: [AIRFLOW_3982] Update DagRun state based on 
its tasks rather than DAG's
URL: https://github.com/apache/airflow/pull/4808#issuecomment-475834939
 
 
   one small nit, LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's

feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update 
DagRun state based on its tasks rather than DAG's
URL: https://github.com/apache/airflow/pull/4808#discussion_r268380301
 
 

 ##
 File path: tests/test_jobs.py
 ##
 @@ -2428,6 +2428,24 @@ def test_dagrun_root_fail_unfinished(self):
 dr_state = dr.update_state()
 self.assertEqual(dr_state, State.RUNNING)
 
+def test_dagrun_root_after_dagrun_unfinished(self):
+"""
+DagRuns with one successful and one future root task -> SUCCESS
+"""
+dag_id = 'test_dagrun_states_root_future'
+dag = self.dagbag.get_dag(dag_id)
+dag.clear()
+scheduler = SchedulerJob(dag_id, num_runs=2)
+# we can't use dag.run or evaluate_dagrun because it uses BackfillJob
+# instead of SchedulerJob and BackfillJobs are allowed to not respect 
start dates
 
 Review comment:
   @dima-asana , WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's

feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update 
DagRun state based on its tasks rather than DAG's
URL: https://github.com/apache/airflow/pull/4808#discussion_r268379986
 
 

 ##
 File path: tests/test_jobs.py
 ##
 @@ -2428,6 +2428,24 @@ def test_dagrun_root_fail_unfinished(self):
 dr_state = dr.update_state()
 self.assertEqual(dr_state, State.RUNNING)
 
+def test_dagrun_root_after_dagrun_unfinished(self):
+"""
+DagRuns with one successful and one future root task -> SUCCESS
+"""
+dag_id = 'test_dagrun_states_root_future'
+dag = self.dagbag.get_dag(dag_id)
+dag.clear()
+scheduler = SchedulerJob(dag_id, num_runs=2)
+# we can't use dag.run or evaluate_dagrun because it uses BackfillJob
+# instead of SchedulerJob and BackfillJobs are allowed to not respect 
start dates
 
 Review comment:
   nit: you mean `Unlike SchedulerJob, BackfillJobs are allowed to not respect 
start dates` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update DagRun state based on its tasks rather than DAG's

feng-tao commented on a change in pull request #4808: [AIRFLOW_3982] Update 
DagRun state based on its tasks rather than DAG's
URL: https://github.com/apache/airflow/pull/4808#discussion_r268379986
 
 

 ##
 File path: tests/test_jobs.py
 ##
 @@ -2428,6 +2428,24 @@ def test_dagrun_root_fail_unfinished(self):
 dr_state = dr.update_state()
 self.assertEqual(dr_state, State.RUNNING)
 
+def test_dagrun_root_after_dagrun_unfinished(self):
+"""
+DagRuns with one successful and one future root task -> SUCCESS
+"""
+dag_id = 'test_dagrun_states_root_future'
+dag = self.dagbag.get_dag(dag_id)
+dag.clear()
+scheduler = SchedulerJob(dag_id, num_runs=2)
+# we can't use dag.run or evaluate_dagrun because it uses BackfillJob
+# instead of SchedulerJob and BackfillJobs are allowed to not respect 
start dates
 
 Review comment:
   nit: you mean `unlike SchedulerJob and BackfillJobs are allowed to not 
respect start dates` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on issue #4949: [AIRFLOW-1557] Backfill should respect pool limit

milton0825 commented on issue #4949: [AIRFLOW-1557] Backfill should respect 
pool limit
URL: https://github.com/apache/airflow/pull/4949#issuecomment-475834272
 
 
   @ashb @feng-tao ready for another look


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268379885
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   I use 
https://github.com/apache/airflow/pull/4895/files#diff-0ddca7f4132e6c4a5dc1ee70d4008085R108
 overwrite default value if user set specific value in connection `host`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268379801
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   That a good idea.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes

pgagnon commented on issue #4648: [AIRFLOW-3274] Add run_as_user and fs_group 
options for Kubernetes
URL: https://github.com/apache/airflow/pull/4648#issuecomment-475830932
 
 
   @dimberman @ashb From what I understand the window for 1.10.3 is closing 
soon. Do you think we can get this in? I am available to work on any further 
change requests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268376225
 
 

 ##
 File path: airflow/config_templates/default_test.cfg
 ##
 @@ -48,6 +48,7 @@ dag_concurrency = 16
 dags_are_paused_at_creation = False
 fernet_key = {FERNET_KEY}
 non_pooled_task_slot_count = 128
+non_pooled_backfill_task_slot_count = 128
 
 Review comment:
   fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268375067
 
 

 ##
 File path: airflow/jobs.py
 ##
 @@ -2300,8 +2301,33 @@ def _per_task_process(task, key, ti, session=None):
 self.log.debug('Adding %s to not_ready', ti)
 ti_status.not_ready.add(key)
 
+pool_name = self.pool
+if self.pool:
+pool = session.query(models.Pool) \
+.filter(models.Pool.pool == self.pool) \
+.first()
+if pool is not None:
+open_slots = pool.open_slots(session=session)
+else:
+open_slots = 0
+self.log.warning(
+"Tasks using non-existent pool '%s' will not be 
scheduled",
+pool,
+)
+else:
+open_slots = conf.getint('core', 
'non_pooled_backfill_task_slot_count')
+pool_name = 'not_pooled'
+
 for task in self.dag.topological_sort():
 for key, ti in list(ti_status.to_run.items()):
+if open_slots <= 0:
+self.log.debug(
+"Not scheduling since there are %s open slots in 
pool %s",
+open_slots, pool_name,
+)
+break
+open_slots -= 1
 
 Review comment:
   Backfilling would not stop actually. We will wait here for later retry:
   
https://github.com/apache/airflow/blob/84a7a9e6cb81139a2b62996acbd8579aff576810/airflow/jobs.py#L2484-L2508
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #4900: [AIRFLOW-3362] [WIP]Template to support jinja2 native python types

mik-laj commented on issue #4900: [AIRFLOW-3362] [WIP]Template to support 
jinja2 native python types
URL: https://github.com/apache/airflow/pull/4900#issuecomment-475824381
 
 
   Have you analyzed the use of literal_eval? It is currently used in some 
operators
   
https://github.com/apache/airflow/blob/2c255e9e0cb8c42c52b2db4c3f863d977ca0e048/airflow/contrib/operators/jenkins_job_trigger_operator.py#L138-L139
   
https://github.com/apache/airflow/blob/067a1e3f4aa153195674950fee72bdd23652e61d/airflow/operators/docker_operator.py#L244-L248


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268365670
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   One option could be to just override `get_conn()` in our class to be 
hard-coded:
   
   ```python
   def get_conn(self, headers=None):
   session = requests.Session()
   if headers:
   session.headers.update(headers)
   return session
   ```
   
   Since we already set base_url else where, and we don't want to support 
generic params (login/password, extra headers) this might actually be the best 
approach?
   
   What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268365670
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   One option could be to just override `get_conn()` in our class to be 
hard-coded:
   
   ```
   def get_conn(self):
  return requests.Session()
   ```
   
   Since we already set base_url else where, and we don't want to support 
generic params (login/password, extra headers) this might actually be the best 
approach?
   
   What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268365342
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   Oh but yes, I mis-read the code in http_hook. So my approach won't work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268364976
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   ```
   In [1]: from airflow.models.connection import Connection
   
   In [2]: c = Connection()
   
   In [3]: c.parse_from_uri("//:password@")
   
   In [4]: c.host
   Out[4]: ''
   
   In [5]: c.host is None
   Out[5]: False
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME

ashb commented on issue #4705: [AIRFLOW-3743] Unify different methods of 
working out AIRFLOW_HOME
URL: https://github.com/apache/airflow/pull/4705#issuecomment-475809406
 
 
   @XD-DENG PTAnotherL :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4705: [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME

ashb commented on a change in pull request #4705: [AIRFLOW-3743] Unify 
different methods of working out AIRFLOW_HOME
URL: https://github.com/apache/airflow/pull/4705#discussion_r268362887
 
 

 ##
 File path: airflow/configuration.py
 ##
 @@ -530,6 +530,24 @@ def parameterized_config(template):
 
 conf.read(AIRFLOW_CONFIG)
 
+if conf.has_option('core', 'AIRFLOW_HOME'):
+if 'AIRFLOW_HOME' in os.environ:
+warnings.warn(
+'Specifying both AIRFLOW_HOME environment variable and 
airflow_home '
+'in the config file is deprecated. Please use only the 
AIRFLOW_HOME '
+'environment variable and remove the config file entry.',
+category=DeprecationWarning,
+)
+else:
+AIRFLOW_HOME = conf.get('core', 'airflow_home')
 
 Review comment:
   I've changed it to:
   
   ```python
   if conf.has_option('core', 'AIRFLOW_HOME'):
   msg = (
   'Specifying both AIRFLOW_HOME environment variable and airflow_home '
   'in the config file is deprecated. Please use only the AIRFLOW_HOME '
   'environment variable and remove the config file entry.'
   )
   if 'AIRFLOW_HOME' in os.environ:
   warnings.warn(msg, category=DeprecationWarning,)
   elif conf.get('core', 'airflow_home') == AIRFLOW_HOME:
   warnings.warn(
   'Specifying airflow_home in the config file is deprecated. As 
you '
   'have left it at the default value you should remove the setting 
'
   'from your airflow.cfg and suffer no change in behaviour.',
   category=DeprecationWarning,
   )
   else:
   AIRFLOW_HOME = conf.get('core', 'airflow_home')
   warnings.warn(msg, category=DeprecationWarning,)
   
   ```
   
   I   thought about adding an extra case do say which value it was


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4146) CgroupTaskRunner is not functioning

2019-03-22 Thread Junda Yang (JIRA)

Junda Yang created AIRFLOW-4146:
---

 Summary: CgroupTaskRunner is not functioning
 Key: AIRFLOW-4146
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4146
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Junda Yang
Assignee: Junda Yang


Tried to switch from StandardTaskRunner to CgroupTaskRunner but tasks were 
stuck and unable to finish. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #4961: [AIRFLOW-XXX] Add Daniel to the committer list

ashb merged pull request #4961: [AIRFLOW-XXX] Add Daniel to the committer list
URL: https://github.com/apache/airflow/pull/4961
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #4961: [AIRFLOW-XXX] Add Daniel to the committer list

feng-tao commented on issue #4961: [AIRFLOW-XXX] Add Daniel to the committer 
list
URL: https://github.com/apache/airflow/pull/4961#issuecomment-475805613
 
 
   cc @XD-DENG @dimberman 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao opened a new pull request #4961: [AIRFLOW-XXX] Add Daniel to the committer list

feng-tao opened a new pull request #4961: [AIRFLOW-XXX] Add Daniel to the 
committer list
URL: https://github.com/apache/airflow/pull/4961
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor

mans2singh commented on issue #4887: [AIRFLOW-4055] Add AWS SQS Sensor
URL: https://github.com/apache/airflow/pull/4887#issuecomment-475799403
 
 
   @mik-laj @zhongjiajie @feng-tao team - Please let me know if you have any 
additional comments for me. Thanks for your advice.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable

codecov-io edited a comment on issue #4960: [AIRFLOW-4145] Allow RBAC roles 
permissions , VM to be overridable
URL: https://github.com/apache/airflow/pull/4960#issuecomment-475790381
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=h1) 
Report
   > Merging 
[#4960](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/40acf579bcd8d2538a68e9bd86041048dbc94318?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4960/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4960  +/-   ##
   =
   + Coverage75.6%   75.6%   +<.01% 
   =
 Files 455 455  
 Lines   29467   29467  
   =
   + Hits22278   22279   +1 
   + Misses   71897188   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4960/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `92.3% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4960/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.92% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=footer). 
Last update 
[40acf57...69c2974](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable

codecov-io commented on issue #4960: [AIRFLOW-4145] Allow RBAC roles 
permissions , VM to be overridable
URL: https://github.com/apache/airflow/pull/4960#issuecomment-475790381
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=h1) 
Report
   > Merging 
[#4960](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=desc) into 
[master](https://codecov.io/gh/apache/airflow/commit/40acf579bcd8d2538a68e9bd86041048dbc94318?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4960/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4960  +/-   ##
   =
   + Coverage75.6%   75.6%   +<.01% 
   =
 Files 455 455  
 Lines   29467   29467  
   =
   + Hits22278   22279   +1 
   + Misses   71897188   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/security.py](https://codecov.io/gh/apache/airflow/pull/4960/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvc2VjdXJpdHkucHk=)
 | `92.3% <100%> (ø)` | :arrow_up: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/4960/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.92% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=footer). 
Last update 
[40acf57...69c2974](https://codecov.io/gh/apache/airflow/pull/4960?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799271#comment-16799271
 ] 

Tao Feng edited comment on AIRFLOW-4139 at 3/22/19 7:07 PM:


there is a long pr([https://github.com/apache/airflow/pull/4396)] from the 
user. I believe the graph_id is the version proposed by the AIP which the main 
goal to solve the dag shape change problem. I would suggest you follow up with 
author or mailing list to avoid any duplicate  effort. The scope of his AIP and 
yours may not be totally aligned, but I feel there is certain overlap between 
these two which I suggest you to clarify with him or the mailing list first.


was (Author: taofeng):
there is a long pr([https://github.com/apache/airflow/pull/4396)] from the 
user. I believe the graph_id is the version proposed by the AIP which the main 
goal to solve the dag shape change problem. I would suggest you follow up with 
author or mailing list to avoid any duplicate  effort.

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
> and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
> ready, we can start saving each version of the DAG file on the remote system 
> and the running tasks should refer to a specific version of DAG instead of 
> the latest DAG.
> How is it different from 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?
> Please see 
> https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4139) DAG versioning



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao-Han Tsai updated AIRFLOW-4139:
---
Description: Currently, running DagRun will be impacted if we change the 
DAG file in the middle of the run. After we have   (was: Currently, running 
DagRun will be impacted if we change the DAG file. Existing running DagRun 
should not be impacted.)

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799271#comment-16799271
 ] 

Tao Feng commented on AIRFLOW-4139:
---

there is a long pr([https://github.com/apache/airflow/pull/4396)] from the 
user. I believe the graph_id is the version proposed by the AIP which the main 
goal to solve the dag shape change problem. I would suggest you follow up with 
author or mailing list to avoid any duplicate  effort.

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
> and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
> ready, we can start saving each version of the DAG file on the remote system 
> and the running tasks should refer to a specific version of DAG instead of 
> the latest DAG.
> How is it different from 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?
> Please see 
> https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799264#comment-16799264
 ] 

Chao-Han Tsai commented on AIRFLOW-4139:


I took a quick scan of 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB 
and I think that they are only persisting DAG graphs for each DAG version in 
the metastore just for the purpose of making webserver stateless. That is not 
enough information to run a task at a given DAG version. We will need to 
version the code in a way that Airflow worker can refer to.

What I am proposing here is that once we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready. We will have the DAG file stored in a versioned storage. Each running 
TaskInstance would run specific version of code.

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file. 
> Existing running DagRun should not be impacted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4139) DAG versioning



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao-Han Tsai updated AIRFLOW-4139:
---
Description: 
Currently, running DagRun will be impacted if we change the DAG file in the 
middle of the run. After we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready, we can start saving each version of the DAG file on the remote system 
and the running tasks should refer to a specific version of DAG instead of the 
latest DAG.

How is it different from 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?
Please see 
https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264

  was:
Currently, running DagRun will be impacted if we change the DAG file in the 
middle of the run. After we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready, we can start saving each version of the DAG file on the remote system 
and the running tasks should refer to a specific version of DAG instead of the 
latest DAG.

How is it different from 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?



> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
> and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
> ready, we can start saving each version of the DAG file on the remote system 
> and the running tasks should refer to a specific version of DAG instead of 
> the latest DAG.
> How is it different from 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?
> Please see 
> https://issues.apache.org/jira/browse/AIRFLOW-4139?focusedCommentId=16799264=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16799264



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4139) DAG versioning



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao-Han Tsai updated AIRFLOW-4139:
---
Description: 
Currently, running DagRun will be impacted if we change the DAG file in the 
middle of the run. After we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready, we can start saving each version of the DAG file on the remote system 
and the running tasks should refer to a specific version of DAG instead of the 
latest DAG.

How is it different from 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?


  was:
Currently, running DagRun will be impacted if we change the DAG file in the 
middle of the run. After we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready, we can start saving each version of the DAG file on the remote system 
and the running tasks should refer to a specific version of DAG instead of the 
latest DAG.

How is it different from 


> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
> and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
> ready, we can start saving each version of the DAG file on the remote system 
> and the running tasks should refer to a specific version of DAG instead of 
> the latest DAG.
> How is it different from 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4139) DAG versioning



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao-Han Tsai updated AIRFLOW-4139:
---
Description: 
Currently, running DagRun will be impacted if we change the DAG file in the 
middle of the run. After we have 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
ready, we can start saving each version of the DAG file on the remote system 
and the running tasks should refer to a specific version of DAG instead of the 
latest DAG.

How is it different from 

  was:Currently, running DagRun will be impacted if we change the DAG file in 
the middle of the run. After we have 


> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file in the 
> middle of the run. After we have 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher 
> and https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest 
> ready, we can start saving each version of the DAG file on the remote system 
> and the running tasks should refer to a specific version of DAG instead of 
> the latest DAG.
> How is it different from 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799265#comment-16799265
 ] 

Chao-Han Tsai commented on AIRFLOW-4139:


I think I should better add more description as it must be very confusing of 
what we aim for this ticket.

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file. 
> Existing running DagRun should not be impacted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268292843
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   I try some time. And find we could not just set `self.base_url = 
"https://oapi.dingtalk.com"` in here.
   If `conn.host` is None, 
   
https://github.com/apache/airflow/blob/40acf579bcd8d2538a68e9bd86041048dbc94318/airflow/hooks/http_hook.py#L63
   will raise error.
   So I add describe in hook and `hwoto` use about we must add `host` to 
`dingding_default`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268291624
 
 

 ##
 File path: airflow/utils/db.py
 ##
 @@ -286,6 +286,13 @@ def initdb():
 Connection(
 conn_id='cassandra_default', conn_type='cassandra',
 host='cassandra', port=9042))
+merge_conn(
+Connection(
+conn_id='dingding_webhook_default', conn_type='http',
+host='https://oapi.dingtalk.com',
 
 Review comment:
   Fix all but except 
https://github.com/apache/airflow/pull/4895#discussion_r268210584, I don't kwon 
what you mean in there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268291701
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
 
 Review comment:
   Fix


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268290966
 
 

 ##
 File path: airflow/contrib/operators/dingding_webhook_operator.py
 ##
 @@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.dingding_webhook_hook import DingdingWebhookHook
+from airflow.operators.bash_operator import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class DingdingWebhookOperator(BaseOperator):
+"""
+This operator allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
 
 Review comment:
   Fix, and change get token from password


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

zhongjiajie commented on a change in pull request #4895: [AIRFLOW-1526] Add 
dingding hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268290786
 
 

 ##
 File path: airflow/contrib/operators/dingding_webhook_operator.py
 ##
 @@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.dingding_webhook_hook import DingdingWebhookHook
+from airflow.operators.bash_operator import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class DingdingWebhookOperator(BaseOperator):
+"""
+This operator allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding (templated)
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of
 
 Review comment:
   fix


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] feng-tao opened a new pull request #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable

feng-tao opened a new pull request #4960: [AIRFLOW-4145] Allow RBAC roles
permissions , VM to be overridable
URL: https://github.com/apache/airflow/pull/4960

Make sure you have checked _all_ steps below.

### Jira

- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4145
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).

### Description

- [x] Here are some details about my PR, including screenshots of any UI
changes:
Currently all the vm/permissions are static defined in
AirflowSecurityManager. We should setup it up in a way which allows the
subclass security manager could override this settings . And it is a good
practice for every company build their own security manager by subclassing the
AirflowSecurityManager.

### Tests

- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Yes
### Commits

- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release

### Code Quality

- [ ] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[GitHub] [airflow] feng-tao commented on issue #4960: [AIRFLOW-4145] Allow RBAC roles permissions , VM to be overridable

feng-tao commented on issue #4960: [AIRFLOW-4145] Allow RBAC roles permissions 
, VM to be overridable
URL: https://github.com/apache/airflow/pull/4960#issuecomment-475729174
 
 
   PTAL @ashb @XD-DENG 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4145) Allow RBAC roles permissions , VM to be overridable



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799236#comment-16799236
 ] 

ASF GitHub Bot commented on AIRFLOW-4145:
-

feng-tao commented on pull request #4960: [AIRFLOW-4145] Allow RBAC roles 
permissions , VM to be overridable
URL: https://github.com/apache/airflow/pull/4960
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-4145
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
 - In case you are proposing a fundamental code change, you need to create 
an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
 - In case you are adding a dependency, check if the license complies with 
the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Currently all the vm/permissions are static defined in 
AirflowSecurityManager. We should setup it up in a way which allows the 
subclass security manager could override this settings . And it is a good 
practice for every company build their own security manager by subclassing the 
AirflowSecurityManager.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Yes
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
 - If you implement backwards incompatible changes, please leave a note in 
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so 
we can assign it to a appropriate release
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow RBAC roles permissions , VM to be overridable
> ---
>
> Key: AIRFLOW-4145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4145
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Tao Feng
>Assignee: Tao Feng
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-4145) Allow RBAC roles permissions , VM to be overridable

Tao Feng created AIRFLOW-4145:
-

 Summary: Allow RBAC roles permissions , VM to be overridable
 Key: AIRFLOW-4145
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4145
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Tao Feng
Assignee: Tao Feng






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] mgaggero commented on issue #4852: [AIRFLOW-3152] Implements init-containers in Kubernetes Pod Operator

mgaggero commented on issue #4852: [AIRFLOW-3152] Implements init-containers in 
Kubernetes Pod Operator
URL: https://github.com/apache/airflow/pull/4852#issuecomment-475726605
 
 
   > @mgaggero why are you PRing this against the stable branch? Also could you 
please add a test?
   
   I wrote this PR to be used in the official Helm chart that in turn uses an 
airflow container based on Airflow 1.10.2.
   Actually I have tried the code in the master branch but I was unable to go 
beyond the Airflow login when running on k8s with Helm.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] bnutt commented on issue #4627: [AIRFLOW-3258] K8S executor environment variables section.

bnutt commented on issue #4627: [AIRFLOW-3258] K8S executor environment 
variables section.
URL: https://github.com/apache/airflow/pull/4627#issuecomment-475725141
 
 
   Has anyone had an issue with using this functionality by setting environment 
variables for the scheduler?
   
   I am trying to set these variables via environment variable, not directly 
through airflow.cfg, this is because I am using the helm chart and its method 
of setting environment variables is through 
[config](https://github.com/helm/charts/blob/master/stable/airflow/values.yaml#L118)
   
   So I have something like:
   ```
   airflow:
 config: {
 AIRFLOW__KUBERNETES_ENVIRONMENT_VARIABLES: {
 AIRFLOW__CORE__LOGGING_LEVEL: "DEBUG"
   }
 }
   ```
   Also have tried
   ```
   airflow:
 config: {
 AIRFLOW__KUBERNETES__ENVIRONMENT__VARIABLES: {
 AIRFLOW__CORE__LOGGING_LEVEL: "DEBUG" 
   }
 }
   ```
   
   However, when a task comes up for that pod, I do not see the environment 
variables being set for the worker pod. 
   
   Is it not possible to set these variables via environment variable and only 
through  an airflow.cfg file?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mgaggero commented on a change in pull request #4852: [AIRFLOW-3152] Implements init-containers in Kubernetes Pod Operator

mgaggero commented on a change in pull request #4852: [AIRFLOW-3152] Implements 
init-containers in Kubernetes Pod Operator
URL: https://github.com/apache/airflow/pull/4852#discussion_r268282837
 
 

 ##
 File path: airflow/contrib/kubernetes/init_container.py
 ##
 @@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+class InitContainer:
 
 Review comment:
   I also agree with this. In this PR my aim is just to provide the 
init-container class to the Kubernetes Pod Operator with the little impact as 
possible on the existing code. The use of Kubernetes in Airflow has great 
potentiality and a broader K8S support is certainly desirable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799193#comment-16799193
 ] 

Tao Feng commented on AIRFLOW-4139:
---

[~milton0825], I wonder what is the difference between this vs 
[https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-12+Persist+DAG+into+DB]
 as they all try to maintain the DAG version .

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file. 
> Existing running DagRun should not be impacted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4139) DAG versioning



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799194#comment-16799194
 ] 

Tao Feng commented on AIRFLOW-4139:
---

For AIP12, there are couples of people that have been working on the feature 
with pr review already.

> DAG versioning
> --
>
> Key: AIRFLOW-4139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4139
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently, running DagRun will be impacted if we change the DAG file. 
> Existing running DagRun should not be impacted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] feng-tao commented on issue #4949: [AIRFLOW-1557] Backfill should respect pool limit

feng-tao commented on issue #4949: [AIRFLOW-1557] Backfill should respect pool 
limit
URL: https://github.com/apache/airflow/pull/4949#issuecomment-475699583
 
 
   @milton0825 , I kinda agree with @ashb . I think pooling is to use for limit 
resources for different teams which having a different pool for schedule and 
backfill job will be confusing. This will cause the issue that doubling the 
capacity  as @ashb mentioned.   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill should respect pool limit

2019-03-22 Thread ASF subversion and git services (JIRA)

milton0825 commented on a change in pull request #4949: [AIRFLOW-1557] Backfill 
should respect pool limit
URL: https://github.com/apache/airflow/pull/4949#discussion_r268257800
 
 

 ##
 File path: airflow/config_templates/default_airflow.cfg
 ##
 @@ -125,6 +125,10 @@ dags_are_paused_at_creation = True
 # whose size is guided by this config element
 non_pooled_task_slot_count = 128
 
+# When not using pools, the number of backfill tasks per backfill
+# is limited by this config element
+non_pooled_backfill_task_slot_count = 128
 
 Review comment:
   @ashb Let me rephrase it a bit. In some cases, people do not run backfills 
on where scheduler runs...
   
   Anyway I think we should still add this config so that people can fine tune 
the slot count if they launch backfill job on a machine other than scheduler


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-4144) add description of is_delete_operator_pod

2019-03-22 Thread Daniel Imberman (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Imberman resolved AIRFLOW-4144.
--
Resolution: Fixed

> add description of is_delete_operator_pod
> -
>
> Key: AIRFLOW-4144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Daniel Imberman
>Priority: Trivial
> Fix For: 1.10.3
>
>
> h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4144) add description of is_delete_operator_pod



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799173#comment-16799173
 ] 

ASF subversion and git services commented on AIRFLOW-4144:
--

Commit 40acf579bcd8d2538a68e9bd86041048dbc94318 in airflow's branch 
refs/heads/master from marengaz
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=40acf57 ]

[AIRFLOW-4144] add description of is_delete_operator_pod (#4943)

[AIRFLOW-4144] add description of is_delete_operator_pod

> add description of is_delete_operator_pod
> -
>
> Key: AIRFLOW-4144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Daniel Imberman
>Priority: Trivial
> Fix For: 1.10.3
>
>
> h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-4144) add description of is_delete_operator_pod

2019-03-22 Thread Daniel Imberman (JIRA)

Daniel Imberman created AIRFLOW-4144:


 Summary: add description of is_delete_operator_pod
 Key: AIRFLOW-4144
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
 Project: Apache Airflow
  Issue Type: Task
Reporter: Daniel Imberman


h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4144) add description of is_delete_operator_pod

2019-03-22 Thread Daniel Imberman (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Imberman updated AIRFLOW-4144:
-
Fix Version/s: 1.10.3

> add description of is_delete_operator_pod
> -
>
> Key: AIRFLOW-4144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Daniel Imberman
>Priority: Trivial
> Fix For: 1.10.3
>
>
> h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4144) add description of is_delete_operator_pod

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799172#comment-16799172
 ] 

ASF subversion and git services commented on AIRFLOW-4144:
--

Commit 40acf579bcd8d2538a68e9bd86041048dbc94318 in airflow's branch 
refs/heads/master from marengaz
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=40acf57 ]

[AIRFLOW-4144] add description of is_delete_operator_pod (#4943)

[AIRFLOW-4144] add description of is_delete_operator_pod

> add description of is_delete_operator_pod
> -
>
> Key: AIRFLOW-4144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Daniel Imberman
>Priority: Trivial
> Fix For: 1.10.3
>
>
> h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4144) add description of is_delete_operator_pod



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799171#comment-16799171
 ] 

ASF GitHub Bot commented on AIRFLOW-4144:
-

dimberman commented on pull request #4943: [AIRFLOW-4144] add description of 
is_delete_operator_pod
URL: https://github.com/apache/airflow/pull/4943
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> add description of is_delete_operator_pod
> -
>
> Key: AIRFLOW-4144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4144
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Daniel Imberman
>Priority: Trivial
>
> h1. add description of is_delete_operator_pod for the kubernetesoperator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] dimberman merged pull request #4943: [AIRFLOW-4144] add description of is_delete_operator_pod

dimberman merged pull request #4943: [AIRFLOW-4144] add description of 
is_delete_operator_pod
URL: https://github.com/apache/airflow/pull/4943
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-4143) Airflow unaware of worker pods being terminated or failing without being able for Local Executor service updating the task state.

2019-03-22 Thread Paul Bramhall (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Bramhall reassigned AIRFLOW-4143:
--

Assignee: Paul Bramhall

> Airflow unaware of worker pods being terminated or failing without being able 
> for Local Executor service updating the task state.
> -
>
> Key: AIRFLOW-4143
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4143
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: kubernetes
>Affects Versions: 1.10.2
> Environment: Kubernetes, Centos
>Reporter: Paul Bramhall
>Assignee: Paul Bramhall
>Priority: Major
>
> Whenever a worker pod is terminated within Kubernetes, in a way that causes 
> the LocalExecutor running on that pod to exit without being able to update 
> the task state within airflow, AIrflow is unaware and blissfully keeps the 
> task in a running state.  This can prevent future runs of the dag from 
> triggering, or the dag from retrying upon failure.
> When Airflow is restarted, the task state is still marked as running & is not 
> updated.
> There is a JobWatcher process within the Kubernetes Executor, but this has no 
> impact upon this condition.  It will receive a Pod Event of DELETED, and 
> still display the task state as 'Running':
> {code:java}
> [2019-03-20 15:37:54,764] {{kubernetes_executor.py:296}} INFO - Event: 
> kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 had an event 
> of type MODIFIED [2019-03-20 15:37:54,764] {{kubernetes_executor.py:336}} 
> INFO - Event: kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 
> is Running [2019-03-20 15:37:54,767] {{kubernetes_executor.py:296}} INFO - 
> Event: kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 had an 
> event of type DELETED [2019-03-20 15:37:54,767] 
> {{kubernetes_executor.py:336}} INFO - Event: 
> kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 is Running
> {code}
> ^^ This event is if the pod is killed without the Local Executor running on 
> that worker updating the state.  Airflow continues to show the task as 
> Running.
> Ideally we need a check within the Kubernetes Executor to perform the 
> following actions:
> Upon Executor Startup:
>  # Loop through all running Dags and Task Instances
>  # Check for the existence of worker pods associated to these instances.
>  # If no pod is found, yet the task is shown as 'Running', perform a graceful 
> failure using the 'handle_failure()' function within Airflow itself.
> During the executor running, we can perform the same, if not a similar check 
> from within the KubernetesJobWatcher class when the following conditions are 
> met:
>  * k8s Pod Event type is 'DELETED' _and_ Task State is still set as 'Running'
> This way, if a pod does unexpectedly terminate (for example, it exceeds the 
> predefined resource limits and oom's), then the dag should be marked as 
> failed accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] marengaz commented on a change in pull request #4943: [AIRFLOW-XXX] add description of is_delete_operator_pod

2019-03-22 Thread ASF subversion and git services (JIRA)

marengaz commented on a change in pull request #4943: [AIRFLOW-XXX] add 
description of is_delete_operator_pod
URL: https://github.com/apache/airflow/pull/4943#discussion_r268237746
 
 

 ##
 File path: airflow/contrib/operators/kubernetes_pod_operator.py
 ##
 @@ -80,6 +80,10 @@ class KubernetesPodOperator(BaseOperator):
 /airflow/xcom/return.json in the container will also be pushed to an
 XCom when the container completes.
 :type do_xcom_push: bool
+:param is_delete_operator_pod: If True, delete the pod after it's reached
+a final state (independent of pod success), or the task instance gets 
 
 Review comment:
   rephrased for clarity, as that is not my intended meaning!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-4143) Airflow unaware of worker pods being terminated or failing without being able for Local Executor service updating the task state.

2019-03-22 Thread Paul Bramhall (JIRA)

Paul Bramhall created AIRFLOW-4143:
--

 Summary: Airflow unaware of worker pods being terminated or 
failing without being able for Local Executor service updating the task state.
 Key: AIRFLOW-4143
 URL: https://issues.apache.org/jira/browse/AIRFLOW-4143
 Project: Apache Airflow
  Issue Type: Bug
  Components: kubernetes
Affects Versions: 1.10.2
 Environment: Kubernetes, Centos
Reporter: Paul Bramhall


Whenever a worker pod is terminated within Kubernetes, in a way that causes the 
LocalExecutor running on that pod to exit without being able to update the task 
state within airflow, AIrflow is unaware and blissfully keeps the task in a 
running state.  This can prevent future runs of the dag from triggering, or the 
dag from retrying upon failure.

When Airflow is restarted, the task state is still marked as running & is not 
updated.

There is a JobWatcher process within the Kubernetes Executor, but this has no 
impact upon this condition.  It will receive a Pod Event of DELETED, and still 
display the task state as 'Running':


{code:java}
[2019-03-20 15:37:54,764] {{kubernetes_executor.py:296}} INFO - Event: 
kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 had an event of 
type MODIFIED [2019-03-20 15:37:54,764] {{kubernetes_executor.py:336}} INFO - 
Event: kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 is 
Running [2019-03-20 15:37:54,767] {{kubernetes_executor.py:296}} INFO - Event: 
kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 had an event of 
type DELETED [2019-03-20 15:37:54,767] {{kubernetes_executor.py:336}} INFO - 
Event: kubernetesexecutorworkeradhocdagworkerpodadhoc-73661b6115-ae484 is 
Running
{code}
^^ This event is if the pod is killed without the Local Executor running on 
that worker updating the state.  Airflow continues to show the task as Running.

Ideally we need a check within the Kubernetes Executor to perform the following 
actions:

Upon Executor Startup:
 # Loop through all running Dags and Task Instances
 # Check for the existence of worker pods associated to these instances.
 # If no pod is found, yet the task is shown as 'Running', perform a graceful 
failure using the 'handle_failure()' function within Airflow itself.

During the executor running, we can perform the same, if not a similar check 
from within the KubernetesJobWatcher class when the following conditions are 
met:
 * k8s Pod Event type is 'DELETED' _and_ Task State is still set as 'Running'

This way, if a pod does unexpectedly terminate (for example, it exceeds the 
predefined resource limits and oom's), then the dag should be marked as failed 
accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-4127) Azure Container Instance Hook _get_instance_view bug

2019-03-22 Thread Xiaodong DENG (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaodong DENG updated AIRFLOW-4127:
---
Fix Version/s: (was: 2.0.0)
   1.10.3

> Azure Container Instance Hook _get_instance_view bug
> 
>
> Key: AIRFLOW-4127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4127
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: 2.0.0
>Reporter: Asgeir Berland
>Assignee: Asgeir Berland
>Priority: Major
>  Labels: easyfix
> Fix For: 1.10.3
>
>   Original Estimate: 20m
>  Remaining Estimate: 20m
>
> Code:
> [https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/azure_container_instance_hook.py#L116]
>  
> The ".current_state" needs to be stripped from the return call in this 
> method. You can easily see why if you look at line107 and line129, which 
> expects an "instance_view" and not "current_state".
> {code}
>  return response.containers[0].instance_view.current_state
> ->  
>  return response.containers[0].instance_view
> {code}
> The current state of this bug makes it impossible to poll containers for 
> status which is essential to detect when they have terminated their jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4002) Optionally start debugger on uncaught exceptions in `airflow test`



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799093#comment-16799093
 ] 

ASF subversion and git services commented on AIRFLOW-4002:
--

Commit b7382f33396cb64dc187438d0dfc8470c104b0a6 in airflow's branch 
refs/heads/v1-10-stable from Joshua Carp
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b7382f3 ]

[AIRFLOW-4002] Option to open debugger on errors in `airflow test`. (#4828)



> Optionally start debugger on uncaught exceptions in `airflow test`
> --
>
> Key: AIRFLOW-4002
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4002
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Josh Carp
>Priority: Trivial
> Fix For: 1.10.3
>
>
> When testing a new task with `airflow test`, it would be useful to optionally 
> open a `pdb` session on uncaught exceptions in order to debug the issue. I 
> propose adding a `–post_mortem` flag to `airflow test` to enable this 
> behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] dimberman commented on issue #4852: [AIRFLOW-3152] Implements init-containers in Kubernetes Pod Operator

dimberman commented on issue #4852: [AIRFLOW-3152] Implements init-containers 
in Kubernetes Pod Operator
URL: https://github.com/apache/airflow/pull/4852#issuecomment-475674348
 
 
   @mgaggero I changed the base branch to master, could you please rebase?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] darrenhaken commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names

darrenhaken commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot 
handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-475666931
 
 
   Thanks so much for all of this! @ashb when do you expect 1.10.3 to be 
released?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #4919: [AIRFLOW-4093] Throw exception if job failed or cancelled or retry too many times

2019-03-22 Thread ASF subversion and git services (JIRA)

codecov-io commented on issue #4919: [AIRFLOW-4093] Throw exception if job 
failed or cancelled or retry too many times
URL: https://github.com/apache/airflow/pull/4919#issuecomment-475666708
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=h1) 
Report
   > :exclamation: No coverage uploaded for pull request base 
(`master@ee71a8b`). [Click here to learn what that 
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/airflow/pull/4919/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ## master#4919   +/-   ##
   =
 Coverage  ?   75.64%   
   =
 Files ?  455   
 Lines ?29450   
 Branches  ?0   
   =
 Hits  ?22278   
 Misses? 7172   
 Partials  ?0
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[airflow/contrib/operators/aws\_athena\_operator.py](https://codecov.io/gh/apache/airflow/pull/4919/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hd3NfYXRoZW5hX29wZXJhdG9yLnB5)
 | `73.46% <100%> (ø)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=footer). 
Last update 
[ee71a8b...232a3d8](https://codecov.io/gh/apache/airflow/pull/4919?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-4140) Pause/unpause button not working

2019-03-22 Thread Ash Berlin-Taylor (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor updated AIRFLOW-4140:
---
Affects Version/s: (was: 1.10.2)

> Pause/unpause button not working
> 
>
> Key: AIRFLOW-4140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4140
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Catarina Silva
>Assignee: Catarina Silva
>Priority: Minor
>
> When using the current version of v1-10-stable, and starting the webserver 
> with the provided scripts in `{color:#33}airflow/scripts/ci/kubernetes` 
> the unpause/pause button does not working. Looking at the logs we see the 
> following error:{color}
> {{{color:#33}[2019-03-21 16:20:11,908] \{app.py:1761} ERROR - Exception 
> on /paused [POST]
> Traceback (most recent call last):
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in 
> wsgi_app
>  response = self.full_dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in 
> full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in 
> handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in 
> full_dispatch_request
>  rv = self.dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in 
> dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 121, in wrapper
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/security/decorators.py",
>  line 26, in wraps
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 56, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 73, 
> in wrapper
>  return func(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/views.py", 
> line 1633, in paused
>  orm_dag.is_paused = False
> AttributeError: 'NoneType' object has no attribute 'is_paused'{color}}}{{}}
>  
> {color:#33}Tracing it back it is possible to check that the request made 
> to change the `is_paused` field of the dag is missing the required `dag_id` 
> field. However this fiel is available for example in the dag link.{color}
> {color:#33}Digging a bit further in the code, we found that this is a bug 
> in the template, which provides the wrong value when building the page.{color}
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-4140) Pause/unpause button not working

2019-03-22 Thread Ash Berlin-Taylor (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-4140.

Resolution: Fixed

Fixed by re-writing the broken commit in v1-10-stable branch as it hasn't yet 
been released.

> Pause/unpause button not working
> 
>
> Key: AIRFLOW-4140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4140
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Reporter: Catarina Silva
>Assignee: Catarina Silva
>Priority: Minor
>
> When using the current version of v1-10-stable, and starting the webserver 
> with the provided scripts in `{color:#33}airflow/scripts/ci/kubernetes` 
> the unpause/pause button does not working. Looking at the logs we see the 
> following error:{color}
> {{{color:#33}[2019-03-21 16:20:11,908] \{app.py:1761} ERROR - Exception 
> on /paused [POST]
> Traceback (most recent call last):
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in 
> wsgi_app
>  response = self.full_dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in 
> full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in 
> handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in 
> full_dispatch_request
>  rv = self.dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in 
> dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 121, in wrapper
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/security/decorators.py",
>  line 26, in wraps
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 56, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 73, 
> in wrapper
>  return func(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/views.py", 
> line 1633, in paused
>  orm_dag.is_paused = False
> AttributeError: 'NoneType' object has no attribute 'is_paused'{color}}}{{}}
>  
> {color:#33}Tracing it back it is possible to check that the request made 
> to change the `is_paused` field of the dag is missing the required `dag_id` 
> field. However this fiel is available for example in the dag link.{color}
> {color:#33}Digging a bit further in the code, we found that this is a bug 
> in the template, which provides the wrong value when building the page.{color}
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3737) Kubernetes executor cannot handle long dag/task names



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799109#comment-16799109
 ] 

ASF subversion and git services commented on AIRFLOW-3737:
--

Commit 5ce9f079708b509cd6f18ed7fc0479591ce0e0d0 in airflow's branch 
refs/heads/v1-10-stable from Paul Bramhall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5ce9f07 ]

[AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636)

Kubernetes has a 63char limit on label values, so when running either a
subdag or a dag with a name with a length longer than 63 chars, the pod
creation process will fail with the following error:

[2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
to parse requirement: invalid label value:
\"very_long_dag_name.very_long_task_name\": must be no more than 63
characters","reason":"BadRequest","code":400}

Annotations however, allow for a relatively unrestricted length when
storing them against pods, and as such would be better suited in this
situation.

Generate deterministic random data for dag_id and task_id
To ensure testing consistency, when calling the '_gen_random_string()'
function, we want to set the seed dependant on the string length and
iteration within the `cases.extend` loop.
This allows random values to be created (as required) but ensuring
consistency of these values during each and every time we run tests
against the code.

> Kubernetes executor cannot handle long dag/task names
> -
>
> Key: AIRFLOW-3737
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3737
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
> Fix For: 1.10.3
>
>
> When using the Kubernetes Executor, if the combined DAG and Task name is 
> longer than 63 characters the scheduler is unable to create a pod and stops 
> processing further dags. For example the logs show the following:
> (edit: our issue appears to be related to using the subdag operator, the 
> string which is 67 chars is the full subdag title)
> {noformat}
> [2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body: 
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
>  to parse requirement: invalid label value: 
> \"very_long_dag_name.very_long_task_name\": must be no more than 63 
> characters","reason":"BadRequest","code":400}
> [2019-01-21 08:07:05,389] {dag_processing.py:579} WARNING - Ending without 
> manager process.
> [2019-01-21 08:07:05,390] {jobs.py:1510} INFO - Exited execute loop
> {noformat}
> The error appears to be returned by the k8s api. 
> Perhaps related to the code here? 
> https://github.com/apache/airflow/blob/a7e369f082025fb089d909a6978106bb7cd24798/airflow/contrib/executors/kubernetes_executor.py#L499-L517
> This was tested using 1.10.2RC2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3737) Kubernetes executor cannot handle long dag/task names

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799108#comment-16799108
 ] 

ASF subversion and git services commented on AIRFLOW-3737:
--

Commit e8c395093d06946d1be5ee072116bc2994dad0a7 in airflow's branch 
refs/heads/master from Paul Bramhall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e8c3950 ]

[AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636)

Kubernetes has a 63char limit on label values, so when running either a
subdag or a dag with a name with a length longer than 63 chars, the pod
creation process will fail with the following error:

[2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body:

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
to parse requirement: invalid label value:
\"very_long_dag_name.very_long_task_name\": must be no more than 63
characters","reason":"BadRequest","code":400}

Annotations however, allow for a relatively unrestricted length when
storing them against pods, and as such would be better suited in this
situation.

Generate deterministic random data for dag_id and task_id
To ensure testing consistency, when calling the '_gen_random_string()'
function, we want to set the seed dependant on the string length and
iteration within the `cases.extend` loop.
This allows random values to be created (as required) but ensuring
consistency of these values during each and every time we run tests
against the code.

> Kubernetes executor cannot handle long dag/task names
> -
>
> Key: AIRFLOW-3737
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3737
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
> Fix For: 1.10.3
>
>
> When using the Kubernetes Executor, if the combined DAG and Task name is 
> longer than 63 characters the scheduler is unable to create a pod and stops 
> processing further dags. For example the logs show the following:
> (edit: our issue appears to be related to using the subdag operator, the 
> string which is 67 chars is the full subdag title)
> {noformat}
> [2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body: 
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
>  to parse requirement: invalid label value: 
> \"very_long_dag_name.very_long_task_name\": must be no more than 63 
> characters","reason":"BadRequest","code":400}
> [2019-01-21 08:07:05,389] {dag_processing.py:579} WARNING - Ending without 
> manager process.
> [2019-01-21 08:07:05,390] {jobs.py:1510} INFO - Exited execute loop
> {noformat}
> The error appears to be returned by the k8s api. 
> Perhaps related to the code here? 
> https://github.com/apache/airflow/blob/a7e369f082025fb089d909a6978106bb7cd24798/airflow/contrib/executors/kubernetes_executor.py#L499-L517
> This was tested using 1.10.2RC2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names

ashb commented on issue #4636: [AIRFLOW-3737] Kubernetes executor cannot handle 
long dag/task names
URL: https://github.com/apache/airflow/pull/4636#issuecomment-475663685
 
 
   Thanks  @PaulW, I'll get  this in 1.10.3  too!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3737) Kubernetes executor cannot handle long dag/task names



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799106#comment-16799106
 ] 

ASF GitHub Bot commented on AIRFLOW-3737:
-

ashb commented on pull request #4636: [AIRFLOW-3737] Kubernetes executor cannot 
handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Kubernetes executor cannot handle long dag/task names
> -
>
> Key: AIRFLOW-3737
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3737
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
> Fix For: 1.10.3
>
>
> When using the Kubernetes Executor, if the combined DAG and Task name is 
> longer than 63 characters the scheduler is unable to create a pod and stops 
> processing further dags. For example the logs show the following:
> (edit: our issue appears to be related to using the subdag operator, the 
> string which is 67 chars is the full subdag title)
> {noformat}
> [2019-01-21 08:07:05,337] {rest.py:219} DEBUG - response body: 
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable
>  to parse requirement: invalid label value: 
> \"very_long_dag_name.very_long_task_name\": must be no more than 63 
> characters","reason":"BadRequest","code":400}
> [2019-01-21 08:07:05,389] {dag_processing.py:579} WARNING - Ending without 
> manager process.
> [2019-01-21 08:07:05,390] {jobs.py:1510} INFO - Exited execute loop
> {noformat}
> The error appears to be returned by the k8s api. 
> Perhaps related to the code here? 
> https://github.com/apache/airflow/blob/a7e369f082025fb089d909a6978106bb7cd24798/airflow/contrib/executors/kubernetes_executor.py#L499-L517
> This was tested using 1.10.2RC2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #4636: [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names

ashb merged pull request #4636: [AIRFLOW-3737] Kubernetes executor cannot 
handle long dag/task names
URL: https://github.com/apache/airflow/pull/4636
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #4959: [AIRFLOW-4142] 'DagRun' has no attribute 'task_id'

2019-03-22 Thread ASF subversion and git services (JIRA)

ashb commented on issue #4959: [AIRFLOW-4142] 'DagRun' has no attribute 
'task_id'
URL: https://github.com/apache/airflow/pull/4959#issuecomment-475662734
 
 
   This leaves DagRun  objects not being deleted - that doesn't sound like the 
right fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3980) Unify logger



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799091#comment-16799091
 ] 

ASF subversion and git services commented on AIRFLOW-3980:
--

Commit f27a63d800f5c4d922802804e2a5224cc5f145b3 in airflow's branch 
refs/heads/v1-10-stable from Kamil Breguła
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f27a63d ]

[AIRFLOW-3980] Unify logger (#4804)


> Unify logger
> 
>
> Key: AIRFLOW-3980
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3980
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Kamil Bregula
>Assignee: Kamil Bregula
>Priority: Trivial
> Fix For: 2.0.0, 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799097#comment-16799097
 ] 

ASF subversion and git services commented on AIRFLOW-4122:
--

Commit c1a23e62f4c92b64208de4029497af5df48bff69 in airflow's branch 
refs/heads/v1-10-stable from Jan Hicken
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c1a23e6 ]

[AIRFLOW-4122] Deprecate chain function (#4940)

* [AIRFLOW-4122] Remove chain function

Bit operation like `>>` or `<<` are suggested
to set dependency, which visual and easier to
explain. and have multiple ways is confusion

* change UPDATING.md as recommend
[ci skip]

This commit differs from 0868070 in that we keep the method but have issued a 
deprecation waring


> Remove chain function in airflow.utils
> --
>
> Key: AIRFLOW-4122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4122
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>  Labels: patch
> Fix For: 1.10.3
>
>
> I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it 
> related to Jira ticket . 
> In PR comment, I was suggest close this PR and submit new one to remove 
> function `chain` in `airflow.utils.heplers`, due to Airflower think use bit 
> operation like `>>` or `<<` visual and easier to explain, and have multiple 
> ways is confusion to users. BTW, this function was created 3 year ago and 
> never get updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4122) Remove chain function in airflow.utils

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799096#comment-16799096
 ] 

ASF subversion and git services commented on AIRFLOW-4122:
--

Commit c1a23e62f4c92b64208de4029497af5df48bff69 in airflow's branch 
refs/heads/v1-10-stable from Jan Hicken
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c1a23e6 ]

[AIRFLOW-4122] Deprecate chain function (#4940)

* [AIRFLOW-4122] Remove chain function

Bit operation like `>>` or `<<` are suggested
to set dependency, which visual and easier to
explain. and have multiple ways is confusion

* change UPDATING.md as recommend
[ci skip]

This commit differs from 0868070 in that we keep the method but have issued a 
deprecation waring


> Remove chain function in airflow.utils
> --
>
> Key: AIRFLOW-4122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4122
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: utils
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>  Labels: patch
> Fix For: 1.10.3
>
>
> I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it 
> related to Jira ticket . 
> In PR comment, I was suggest close this PR and submit new one to remove 
> function `chain` in `airflow.utils.heplers`, due to Airflower think use bit 
> operation like `>>` or `<<` visual and easier to explain, and have multiple 
> ways is confusion to users. BTW, this function was created 3 year ago and 
> never get updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3945) Row of NULLs inserted when not changing permission views

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799100#comment-16799100
 ] 

ASF subversion and git services commented on AIRFLOW-3945:
--

Commit e7f96860f4d8ca72ccc1a47d223f310b938d7a31 in airflow's branch 
refs/heads/v1-10-stable from diederikwp
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e7f9686 ]

[AIRFLOW-3945] Stop inserting row when permission views unchanged (#4764)

- Stop inserting a row where only the id is not NULL in table
  ab_permission_view_role when there are no permission views to update.

- Add a test for above issue


> Row of NULLs inserted when not changing permission views
> 
>
> Key: AIRFLOW-3945
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3945
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.2
>Reporter: Diederik Perdok
>Assignee: Diederik Perdok
>Priority: Minor
> Fix For: 1.10.3
>
>
> When creating new permission views in 
> AirflowSecurityManager.create_custom_dag_permission_view in file 
> airflow/www/security.py, the list of permission views to update might be 
> empty because everything is still up to date. In that case, a row in which 
> only the id is not NULL will still be inserted in table 
> ab_permission_view_role every time.
> This is a minor bug when using PostgreSQL. When using a SQL Server backend, 
> this causes the webserver to crash as SQL Server disallows multiple NULLs in 
> a unique constraint (by default).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4106) Instrument starving tasks in pool

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799098#comment-16799098
 ] 

ASF subversion and git services commented on AIRFLOW-4106:
--

Commit b551776f0ca3da0e9829aae73547595d5496c5f3 in airflow's branch 
refs/heads/v1-10-stable from Chao-Han Tsai
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b551776 ]

[AIRFLOW-4106] instrument staving tasks in pool (#4927)



> Instrument starving tasks in pool
> -
>
> Key: AIRFLOW-4106
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4106
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4107) Instrument executor

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799094#comment-16799094
 ] 

ASF subversion and git services commented on AIRFLOW-4107:
--

Commit 0e73672df6a2b4fb0c0736f62fa5bd287d6e2e16 in airflow's branch 
refs/heads/v1-10-stable from Chao-Han Tsai
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0e73672 ]

[AIRFLOW-4107] instrument executor (#4928)



> Instrument executor
> ---
>
> Key: AIRFLOW-4107
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4107
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Priority: Major
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4123) Add Exception handling for _change_state method in K8 Executor

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799099#comment-16799099
 ] 

ASF subversion and git services commented on AIRFLOW-4123:
--

Commit 34501b81bdfbb3ad3be4ae6fcec3e29613f09677 in airflow's branch 
refs/heads/v1-10-stable from andy-g14
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=34501b8 ]

[AIRFLOW-4123] Add Exception handling for _change_state method in K8 Executor 
(#4941)



> Add Exception handling for _change_state method in K8 Executor
> --
>
> Key: AIRFLOW-4123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4123
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Anand
>Assignee: Anand
>Priority: Minor
> Fix For: 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-4009) Fix docstring issue in GoogleCloudStorageToBigQueryOperator

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799092#comment-16799092
 ] 

ASF subversion and git services commented on AIRFLOW-4009:
--

Commit 09c4d52832000397c6b5918cca9e7f4354591e8b in airflow's branch 
refs/heads/v1-10-stable from Ryan Yuan
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=09c4d52 ]

[AIRFLOW-4009] Fix docstring issue in GCSToBQOperator (#4836)


> Fix docstring issue in GoogleCloudStorageToBigQueryOperator
> ---
>
> Key: AIRFLOW-4009
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4009
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.10.2
>Reporter: Ryan Yuan
>Assignee: Ryan Yuan
>Priority: Trivial
> Fix For: 1.10.3
>
>
> The description for parameter destination_project_dataset_table is incorrect 
> as the format should be 
> {code:java}
> (.|:).
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2568) Implement a Azure Container Instances operator

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-2568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799095#comment-16799095
 ] 

ASF subversion and git services commented on AIRFLOW-2568:
--

Commit 898aec9ce93226066bd499786eb876c832b08f73 in airflow's branch 
refs/heads/v1-10-stable from Omeed Musavi
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=898aec9 ]

[AIRFLOW-2568] Azure Container Instances operator (#4121)

Add an operator to create a Docker container in Azure Container
Instances. Azure Container Instances hosts a container and abstracts
away the infrastructure around orchestration of a container service.

Operator supports creating an ACI container and pull an image from Azure
Container Registry or public Docker registries.


> Implement a Azure Container Instances operator
> --
>
> Key: AIRFLOW-2568
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2568
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Niels Zeilemaker
>Assignee: Niels Zeilemaker
>Priority: Major
> Fix For: 2.0.0, 1.10.3
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268204537
 
 

 ##
 File path: airflow/contrib/operators/dingding_webhook_operator.py
 ##
 @@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.dingding_webhook_hook import DingdingWebhookHook
+from airflow.operators.bash_operator import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class DingdingWebhookOperator(BaseOperator):
+"""
+This operator allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
 
 Review comment:
   I don't think this should be called `http_conn_id` - `dingding_conn_id` 
would be more appropriate.
   
   Additionally how about pulling the token from the password field of the 
connection instead?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268206731
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
 
 Review comment:
   ```suggestion
   super(DingdingWebhookHook, 
self).__init__(http_conn_id=http_conn_id,*args, **kwargs)
   ```
   
   Then we don't need the  next line. (See comment later about changing the 
name of the connection parameter though)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268210865
 
 

 ##
 File path: airflow/utils/db.py
 ##
 @@ -286,6 +286,13 @@ def initdb():
 Connection(
 conn_id='cassandra_default', conn_type='cassandra',
 host='cassandra', port=9042))
+merge_conn(
+Connection(
+conn_id='dingding_webhook_default', conn_type='http',
+host='https://oapi.dingtalk.com',
 
 Review comment:
   Let's remove the `_webhook_` from the name too - that distinction only makes 
sense for slack which has two modes, which we don't have here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268205012
 
 

 ##
 File path: airflow/utils/db.py
 ##
 @@ -286,6 +286,13 @@ def initdb():
 Connection(
 conn_id='cassandra_default', conn_type='cassandra',
 host='cassandra', port=9042))
+merge_conn(
+Connection(
+conn_id='dingding_webhook_default', conn_type='http',
+host='https://oapi.dingtalk.com',
 
 Review comment:
   I would probably  leave the host blank in the default one as most users 
won't need/want to specify it (so the Hook should default to this value, not 
the connection)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268207768
 
 

 ##
 File path: airflow/contrib/hooks/dingding_webhook_hook.py
 ##
 @@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+
+from airflow import AirflowException
+from airflow.hooks.http_hook import HttpHook
+
+
+class DingdingWebhookHook(HttpHook):
+"""
+This hook allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of str
+:param at_all: Should remind all people in group or not. If True, will 
overwrite ``at_mobiles``
+:type at_all: bool
+"""
+
+def __init__(self,
+ http_conn_id='dingding_webhook_default',
+ message_type='text',
+ message=None,
+ at_mobiles=False,
+ at_all=False,
+ *args,
+ **kwargs
+ ):
+super(DingdingWebhookHook, self).__init__(*args, **kwargs)
+self.http_conn_id = http_conn_id
+self.message_type = message_type
+self.message = message
+self.at_mobiles = at_mobiles
+self.at_all = at_all
+
 
 Review comment:
   ```suggestion
   self.base_url = "https://oapi.dingtalk.com;
   
   ```
   
   This will then set the default host if the connection doesn't specify a 
host/path.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268210584
 
 

 ##
 File path: airflow/utils/db.py
 ##
 @@ -286,6 +286,13 @@ def initdb():
 Connection(
 conn_id='cassandra_default', conn_type='cassandra',
 host='cassandra', port=9042))
+merge_conn(
+Connection(
+conn_id='dingding_webhook_default', conn_type='http',
+host='https://oapi.dingtalk.com',
 
 Review comment:
   To specify the password from a URI and to leave the default base_url for the 
host we have to use this a "partial" form such as 
`AIRFLOW_CONE_DINGDING_DEFAULT='//:password@'`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding hook and operator

ashb commented on a change in pull request #4895: [AIRFLOW-1526] Add dingding 
hook and operator
URL: https://github.com/apache/airflow/pull/4895#discussion_r268203954
 
 

 ##
 File path: airflow/contrib/operators/dingding_webhook_operator.py
 ##
 @@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.contrib.hooks.dingding_webhook_hook import DingdingWebhookHook
+from airflow.operators.bash_operator import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class DingdingWebhookOperator(BaseOperator):
+"""
+This operator allows you to post messages to Dingding use webhook.
+Dingding token get from conn_id extra field ``webhook_token`` for security.
+
+For more message detail in
+`Dingding custom bot 
`_
+
+:param http_conn_id: connection that has Dingding ``webhook_token`` in the 
extra field
+:type http_conn_id: str
+:param message_type: Message type you want to send to Dingding
+ support text/link/markdown/actionCard/feedCard five 
types so far
+:type message_type: str
+:param message: The message you want to send on Dingding (templated)
+:type message: str or dict
+:param at_mobiles: Users you want to remind at this message.
+:type at_mobiles: list of
 
 Review comment:
   "list of"? Do you mean `list[str]` perhaps?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-4140) Pause/unpause button not working

2019-03-22 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-4140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799073#comment-16799073
 ] 

ASF GitHub Bot commented on AIRFLOW-4140:
-

ashb commented on pull request #4958: [AIRFLOW-4140] Fix pause/unpause button
URL: https://github.com/apache/airflow/pull/4958
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Pause/unpause button not working
> 
>
> Key: AIRFLOW-4140
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4140
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.10.2
>Reporter: Catarina Silva
>Assignee: Catarina Silva
>Priority: Minor
>
> When using the current version of v1-10-stable, and starting the webserver 
> with the provided scripts in `{color:#33}airflow/scripts/ci/kubernetes` 
> the unpause/pause button does not working. Looking at the logs we see the 
> following error:{color}
> {{{color:#33}[2019-03-21 16:20:11,908] \{app.py:1761} ERROR - Exception 
> on /paused [POST]
> Traceback (most recent call last):
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2292, in 
> wsgi_app
>  response = self.full_dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1815, in 
> full_dispatch_request
>  rv = self.handle_user_exception(e)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1718, in 
> handle_user_exception
>  reraise(exc_type, exc_value, tb)
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in 
> full_dispatch_request
>  rv = self.dispatch_request()
>  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in 
> dispatch_request
>  return self.view_functions[rule.endpoint](**req.view_args)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 121, in wrapper
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/security/decorators.py",
>  line 26, in wraps
>  return f(self, *args, **kwargs)
>  File 
> "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/decorators.py", line 
> 56, in wrapper
>  return f(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 73, 
> in wrapper
>  return func(*args, **kwargs)
>  File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/views.py", 
> line 1633, in paused
>  orm_dag.is_paused = False
> AttributeError: 'NoneType' object has no attribute 'is_paused'{color}}}{{}}
>  
> {color:#33}Tracing it back it is possible to check that the request made 
> to change the `is_paused` field of the dag is missing the required `dag_id` 
> field. However this fiel is available for example in the dag link.{color}
> {color:#33}Digging a bit further in the code, we found that this is a bug 
> in the template, which provides the wrong value when building the page.{color}
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3584) Switch dags index view to orm dags



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799074#comment-16799074
 ] 

ASF subversion and git services commented on AIRFLOW-3584:
--

Commit b6d0351deb513841efcc3a1e930c2b714b53a87b in airflow's branch 
refs/heads/v1-10-stable from CatarinaSilva
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b6d0351 ]

fixup! [AIRFLOW-3584] Use ORM DAGs for index view. (#4390) 



> Switch dags index view to orm dags
> --
>
> Key: AIRFLOW-3584
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3584
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Josh Carp
>Assignee: Josh Carp
>Priority: Minor
> Fix For: 2.0.0
>
>
> Part of AIRFLOW-3562.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [airflow] ashb merged pull request #4958: [AIRFLOW-4140] Fix pause/unpause button

ashb merged pull request #4958: [AIRFLOW-4140] Fix pause/unpause button
URL: https://github.com/apache/airflow/pull/4958
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] bryanyang0528 commented on a change in pull request #4919: [AIRFLOW-4093] Throw exception if job failed or cancelled or retry too many times

bryanyang0528 commented on a change in pull request #4919: [AIRFLOW-4093] Throw 
exception if job failed or cancelled or retry too many times
URL: https://github.com/apache/airflow/pull/4919#discussion_r268200430
 
 

 ##
 File path: airflow/contrib/operators/aws_athena_operator.py
 ##
 @@ -74,7 +78,17 @@ def execute(self, context):
 self.result_configuration['OutputLocation'] = self.output_location
 self.query_execution_id = self.hook.run_query(self.query, 
self.query_execution_context,
   
self.result_configuration, self.client_request_token)
-self.hook.poll_query_status(self.query_execution_id)
+query_status = self.hook.poll_query_status(self.query_execution_id, 
self.max_tries)
+
+if query_status in AWSAthenaHook.FAILURE_STATES:
+raise Exception(
+'Final state of Athena job is {}, query_execution_id is {}.'
+.format(query_status, self.query_execution_id))
+elif not query_status or query_status in 
AWSAthenaHook.INTERMEDIATE_STATES:
+raise Exception(
+'Final state of Athena job is {}. \
+ Max tries of poll status exceeded, query_execution_id is {}.'
 
 Review comment:
   @XD-DENG Thanks for reviewing. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index view.