[jira] [Resolved] (AIRFLOW-3348) Refresh run stats on dag refresh
[ https://issues.apache.org/jira/browse/AIRFLOW-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Szymanski resolved AIRFLOW-3348. --- Resolution: Fixed > Refresh run stats on dag refresh > > > Key: AIRFLOW-3348 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3348 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Marcin Szymanski >Assignee: Marcin Szymanski >Priority: Minor > > In some cases dag run statistics may become outdated, for example, when dag > run is deleted. > As the view from which dag runs are deleted permits to delete runs from > multiple dags in a single transaction, this seems to be the most reasonable > place to update run statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3348) Refresh run stats on dag refresh
[ https://issues.apache.org/jira/browse/AIRFLOW-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Szymanski closed AIRFLOW-3348. - > Refresh run stats on dag refresh > > > Key: AIRFLOW-3348 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3348 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Marcin Szymanski >Assignee: Marcin Szymanski >Priority: Minor > > In some cases dag run statistics may become outdated, for example, when dag > run is deleted. > As the view from which dag runs are deleted permits to delete runs from > multiple dags in a single transaction, this seems to be the most reasonable > place to update run statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3423) Fix unauthenticated access for mongo hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcin Szymanski updated AIRFLOW-3423:
--
Description: Empty field values are stored by Airflow as empty strings not
{{None}}
> Fix unauthenticated access for mongo hook
> -
>
> Key: AIRFLOW-3423
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3423
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.1
>Reporter: Marcin Szymanski
>Assignee: Marcin Szymanski
>Priority: Major
>
> Empty field values are stored by Airflow as empty strings not {{None}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3423) Fix unauthenticated access for mongo hook
Marcin Szymanski created AIRFLOW-3423: - Summary: Fix unauthenticated access for mongo hook Key: AIRFLOW-3423 URL: https://issues.apache.org/jira/browse/AIRFLOW-3423 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.1 Reporter: Marcin Szymanski Assignee: Marcin Szymanski -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3348) Refresh run stats on dag refresh
Marcin Szymanski created AIRFLOW-3348: - Summary: Refresh run stats on dag refresh Key: AIRFLOW-3348 URL: https://issues.apache.org/jira/browse/AIRFLOW-3348 Project: Apache Airflow Issue Type: Improvement Reporter: Marcin Szymanski Assignee: Marcin Szymanski In some cases dag run statistics may become outdated, for example, when dag run is deleted. As the view from which dag runs are deleted permits to delete runs from multiple dags in a single transaction, this seems to be the most reasonable place to update run statistics -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3245) resolve_template_files not processing lists
Marcin Szymanski created AIRFLOW-3245: - Summary: resolve_template_files not processing lists Key: AIRFLOW-3245 URL: https://issues.apache.org/jira/browse/AIRFLOW-3245 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.10.0 Reporter: Marcin Szymanski Assignee: Marcin Szymanski Currently, resolve_template_files does not process template field value if it is a list - only strings are processed. Lists are correctly processed at a further stage of template parsing, and are a common use case for submitting multiple SQL statements -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3218) Support for DAG level poking in external task sensor
Marcin Szymanski created AIRFLOW-3218: - Summary: Support for DAG level poking in external task sensor Key: AIRFLOW-3218 URL: https://issues.apache.org/jira/browse/AIRFLOW-3218 Project: Apache Airflow Issue Type: New Feature Affects Versions: 1.10.0 Reporter: Marcin Szymanski Assignee: Marcin Szymanski Currently external task sensor requires a specific task to finish. This change adds support for DAG level checks, if task ID is not provided -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3065) Scheduler failing tasks when DAG concurrency limit reached
[
https://issues.apache.org/jira/browse/AIRFLOW-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcin Szymanski closed AIRFLOW-3065.
-
Resolution: Duplicate
> Scheduler failing tasks when DAG concurrency limit reached
> --
>
> Key: AIRFLOW-3065
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3065
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
>Reporter: Marcin Szymanski
>Priority: Critical
>
> In a DAG with concurrency limit of 4, with about 150 task inside, when the
> limit of active tasks is reached, the scheduler starts to fail queued tasks.
> They later are retried, but if they have downstream tasks, these remain in
> upstream_failed status.
> A few additional details:
> * celery executor
> * environment upgraded from 1.9 (no issues back then)
> * all configuration in airflow.cfg updated to the latest set of options
> * issue happens both with PyPi 1.10 and a build from branch v1-10-test
> (c36ef06)
>
>
> {noformat}
> [2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
> [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum
> number of running tasks (4) for this task's DAG 'consolidated_db' has been
> reached.
> [2018-09-14 13:51:23,852] {models.py:1531} WARNING -
>
> FIXME: Rescheduling due to concurrency limits reached at task runtime.
> Attempt 1 of 1. State set to NONE.
>
> [2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
> [2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
> [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum
> number of running tasks (4) for this task's DAG 'consolidated_db' has been
> reached.
> [2018-09-14 13:51:23,852] {models.py:1531} WARNING -
>
> FIXME: Rescheduling due to concurrency limits reached at task runtime.
> Attempt 1 of 1. State set to NONE.
>
> [2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
> [2018-09-14 13:52:49,939] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:52:50,142] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:52:50,235] {models.py:1548} INFO -
>
> Starting attempt 1 of 1
>
> [2018-09-14 13:52:50,646] {models.py:1570} INFO - Executing
> on 2018-09-14T12:42:55.379761+00:00
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3065) Scheduler failing tasks when DAG concurrency limit reached
[
https://issues.apache.org/jira/browse/AIRFLOW-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617509#comment-16617509
]
Marcin Szymanski commented on AIRFLOW-3065:
---
Turned out to be caused by AIRFLOW-1104.
@commiters can we have the fix included in 1.10.1?
> Scheduler failing tasks when DAG concurrency limit reached
> --
>
> Key: AIRFLOW-3065
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3065
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.0
>Reporter: Marcin Szymanski
>Priority: Critical
>
> In a DAG with concurrency limit of 4, with about 150 task inside, when the
> limit of active tasks is reached, the scheduler starts to fail queued tasks.
> They later are retried, but if they have downstream tasks, these remain in
> upstream_failed status.
> A few additional details:
> * celery executor
> * environment upgraded from 1.9 (no issues back then)
> * all configuration in airflow.cfg updated to the latest set of options
> * issue happens both with PyPi 1.10 and a build from branch v1-10-test
> (c36ef06)
>
>
> {noformat}
> [2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
> [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum
> number of running tasks (4) for this task's DAG 'consolidated_db' has been
> reached.
> [2018-09-14 13:51:23,852] {models.py:1531} WARNING -
>
> FIXME: Rescheduling due to concurrency limits reached at task runtime.
> Attempt 1 of 1. State set to NONE.
>
> [2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
> [2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
> [queued]>, dependency 'Task Instance Slots Available' FAILED: The maximum
> number of running tasks (4) for this task's DAG 'consolidated_db' has been
> reached.
> [2018-09-14 13:51:23,852] {models.py:1531} WARNING -
>
> FIXME: Rescheduling due to concurrency limits reached at task runtime.
> Attempt 1 of 1. State set to NONE.
>
> [2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
> [2018-09-14 13:52:49,939] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:52:50,142] {models.py:1336} INFO - Dependencies all met for
>
> [2018-09-14 13:52:50,235] {models.py:1548} INFO -
>
> Starting attempt 1 of 1
>
> [2018-09-14 13:52:50,646] {models.py:1570} INFO - Executing
> on 2018-09-14T12:42:55.379761+00:00
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3065) Scheduler failing tasks when DAG concurrency limit reached
[
https://issues.apache.org/jira/browse/AIRFLOW-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcin Szymanski updated AIRFLOW-3065:
--
Description:
In a DAG with concurrency limit of 4, with about 150 task inside, when the
limit of active tasks is reached, the scheduler starts to fail queued tasks.
They later are retried, but if they have downstream tasks, these remain in
upstream_failed status.
A few additional details:
* celery executor
* environment upgraded from 1.9 (no issues back then)
* all configuration in airflow.cfg updated to the latest set of options
* issue happens both with PyPi 1.10 and a build from branch v1-10-test
(c36ef06)
{noformat}
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:52:49,939] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,142] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,235] {models.py:1548} INFO -
Starting attempt 1 of 1
[2018-09-14 13:52:50,646] {models.py:1570} INFO - Executing
on 2018-09-14T12:42:55.379761+00:00
{noformat}
was:
In a DAG with concurrency limit of 4, with about 150 task inside, when the
limit limit of active tasks is reached, the scheduler starts to fail queued
tasks. They later are retried, but if they have downstream tasks, these remain
in upstream_failed status.
A few additional details:
* celery executor
* environment upgraded from 1.9 (no issues back then)
* all configuration in airflow.cfg updated to the latest set of options
* issue happens both with PyPi 1.10 and a build from branch v1-10-test
(c36ef06)
{noformat}
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:52:49,939] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,142] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,235] {models.py:1548} INFO -
Starting attempt 1 of 1
[jira] [Created] (AIRFLOW-3065) Scheduler failing tasks when DAG concurrency limit reached
Marcin Szymanski created AIRFLOW-3065:
-
Summary: Scheduler failing tasks when DAG concurrency limit reached
Key: AIRFLOW-3065
URL: https://issues.apache.org/jira/browse/AIRFLOW-3065
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10.0
Reporter: Marcin Szymanski
In a DAG with concurrency limit of 4, with about 150 task inside, when the
limit limit of active tasks is reached, the scheduler starts to fail queued
tasks. They later are retried, but if they have downstream tasks, these remain
in upstream_failed status.
A few additional details:
* celery executor
* environment upgraded from 1.9 (no issues back then)
* all configuration in airflow.cfg updated to the latest set of options
* issue happens both with PyPi 1.10 and a build from branch v1-10-test
(c36ef06)
{noformat}
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:51:23,560] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:51:23,850] {models.py:1330} INFO - Dependencies not met for
,
dependency 'Task Instance Slots Available' FAILED: The maximum number of
running tasks (4) for this task's DAG 'consolidated_db' has been reached.
[2018-09-14 13:51:23,852] {models.py:1531} WARNING -
FIXME: Rescheduling due to concurrency limits reached at task runtime. Attempt
1 of 1. State set to NONE.
[2018-09-14 13:51:23,853] {models.py:1534} INFO - Queuing into pool None
[2018-09-14 13:52:49,939] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,142] {models.py:1336} INFO - Dependencies all met for
[2018-09-14 13:52:50,235] {models.py:1548} INFO -
Starting attempt 1 of 1
[2018-09-14 13:52:50,646] {models.py:1570} INFO - Executing
on 2018-09-14T12:42:55.379761+00:00
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1979) Redis celery backend not work on 1.9.0 (configuration is ignored)
[
https://issues.apache.org/jira/browse/AIRFLOW-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562389#comment-16562389
]
Marcin Szymanski commented on AIRFLOW-1979:
---
If you migrated your config from 1.8, then check if this is in your config file
{code:java}
celery_config_options =
airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
{code}
> Redis celery backend not work on 1.9.0 (configuration is ignored)
> -
>
> Key: AIRFLOW-1979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1979
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery, worker
>Affects Versions: 1.9.0
>Reporter: Norio Akagi
>Priority: Major
>
> Worker tries to connect to RabbigMQ based on a default setting and shows an
> error as below:
> {noformat}
> [2018-01-09 16:45:42,778] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2018-01-09 16:45:42,802] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2018-01-09 16:45:43,051] {configuration.py:206} WARNING - section/key
> [celery/celery_ssl_active] not found in config
> [2018-01-09 16:45:43,051] {default_celery.py:41} WARNING - Celery Executor
> will run without SSL
> [2018-01-09 16:45:43,052] {__init__.py:45} INFO - Using executor
> CeleryExecutor
> [2018-01-09 16:45:43,140: WARNING/MainProcess]
> /usr/local/lib/python2.7/dist-packages/celery/apps/worker.py:161:
> CDeprecationWarning:
> Starting from version 3.2 Celery will refuse to accept pickle by default.
> The pickle serializer is a security concern as it may give attackers
> the ability to execute any command. It's important to secure
> your broker from unauthorized access when using pickle, so we think
> that enabling pickle should require a deliberate action and not be
> the default choice.
> If you depend on pickle then you should set a setting to disable this
> warning and to be sure that everything will continue working
> when you upgrade to Celery 3.2::
> CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
> You must only enable the serializers that you will actually use.
> warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
> [2018-01-09 16:45:43,240: ERROR/MainProcess] consumer: Cannot connect to
> amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
> Trying again in 2.00 seconds...
> {noformat}
> I deploy Airflow on kubernetes so each component (web, scheduler, worker, and
> flower) is containerized and distributed among nodes. I set
> {{AIRFLOW__CELERY__CELERY_RESULT_BACKEND}}
> and {{AIRFLOW__CELERY__BROKER_URL}} in environment variables and it can be
> seen when I run {{printenv}} in a container, but it looks completely ignored.
> Moving these values to {{airflow.cfg}} doesn't work either.
> It worked just perfectly 1.8 and suddenly stopped working when I upgraded
> Airflow to 1.9.
> Do you have any idea what may cause this configuration issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2377) Improve Sendgrid sender support
Marcin Szymanski created AIRFLOW-2377: - Summary: Improve Sendgrid sender support Key: AIRFLOW-2377 URL: https://issues.apache.org/jira/browse/AIRFLOW-2377 Project: Apache Airflow Issue Type: Improvement Components: contrib Reporter: Marcin Szymanski Assignee: Marcin Szymanski * Add support for for sender name * Allow passing sender email and name via kwargs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2281) Add support for Sendgrid categories
[ https://issues.apache.org/jira/browse/AIRFLOW-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Szymanski updated AIRFLOW-2281: -- Summary: Add support for Sendgrid categories (was: Add support for Sendgrid category) > Add support for Sendgrid categories > --- > > Key: AIRFLOW-2281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2281 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Marcin Szymanski >Assignee: Marcin Szymanski >Priority: Minor > > Allow adding categories to sendgrid mails > https://github.com/sendgrid/sendgrid-python/blob/master/sendgrid/helpers/mail/category.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2281) Add support for Sendgrid category
Marcin Szymanski created AIRFLOW-2281: - Summary: Add support for Sendgrid category Key: AIRFLOW-2281 URL: https://issues.apache.org/jira/browse/AIRFLOW-2281 Project: Apache Airflow Issue Type: New Feature Components: contrib Reporter: Marcin Szymanski Assignee: Marcin Szymanski Allow adding categories to sendgrid mails https://github.com/sendgrid/sendgrid-python/blob/master/sendgrid/helpers/mail/category.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-83) Add MongoDB hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-83?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368727#comment-16368727 ] Marcin Szymanski commented on AIRFLOW-83: - This is cool. I have one extra request. Can you add an additional parameter for a custom callable, and then, if set use the function to transform the result, somewhere around line 88 in mongo_to_s3.py? Comes in useful, when converting a nested mongo structure to something else, before loading to s3. Thanks > Add MongoDB hook and operator > - > > Key: AIRFLOW-83 > URL: https://issues.apache.org/jira/browse/AIRFLOW-83 > Project: Apache Airflow > Issue Type: Wish > Components: contrib, hooks, operators >Reporter: vishal srivastava >Assignee: Ajay Yadava >Priority: Minor > > A mongodb hook and operator will be really useful for people who use airflow > with the mongo database. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2102) Add custom_args to Sendgrid personalizations
Marcin Szymanski created AIRFLOW-2102:
-
Summary: Add custom_args to Sendgrid personalizations
Key: AIRFLOW-2102
URL: https://issues.apache.org/jira/browse/AIRFLOW-2102
Project: Apache Airflow
Issue Type: New Feature
Components: contrib
Reporter: Marcin Szymanski
Assignee: Marcin Szymanski
Add support for {{custom_args}} in personalizations
[https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/personalizations.html]
{{custom_args}} should be passed in {{kwargs}} as other backends don't support
them
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-1958) Add **kwargs to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-1958 started by Marcin Szymanski. - > Add **kwargs to send_email > -- > > Key: AIRFLOW-1958 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1958 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Marcin Szymanski >Assignee: Marcin Szymanski >Priority: Minor > > Additional parameters will can be used with backends other than SMTP. This > also gives greater flexibility when using the same logic just different > backends across environment, for example SMTP in prod, dummy in dev and test -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1958) Add **kwargs to send_email
[ https://issues.apache.org/jira/browse/AIRFLOW-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcin Szymanski reassigned AIRFLOW-1958: - Assignee: Marcin Szymanski > Add **kwargs to send_email > -- > > Key: AIRFLOW-1958 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1958 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Marcin Szymanski >Assignee: Marcin Szymanski >Priority: Minor > > Additional parameters will can be used with backends other than SMTP. This > also gives greater flexibility when using the same logic just different > backends across environment, for example SMTP in prod, dummy in dev and test -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1958) Add **kwargs to send_email
Marcin Szymanski created AIRFLOW-1958: - Summary: Add **kwargs to send_email Key: AIRFLOW-1958 URL: https://issues.apache.org/jira/browse/AIRFLOW-1958 Project: Apache Airflow Issue Type: Improvement Reporter: Marcin Szymanski Priority: Minor Additional parameters will can be used with backends other than SMTP. This also gives greater flexibility when using the same logic just different backends across environment, for example SMTP in prod, dummy in dev and test -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1723) Support sendgrid in email backend
[ https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222943#comment-16222943 ] Marcin Szymanski commented on AIRFLOW-1723: --- Ok, thx > Support sendgrid in email backend > -- > > Key: AIRFLOW-1723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1723 > Project: Apache Airflow > Issue Type: New Feature > Components: utils >Affects Versions: 1.9.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > Fix For: 1.10.0 > > > Current airflow email backend only supports SMTP, this PR aims to extend > email backend and integrate with sendgrid. The airflow config file is also > updated to include a [sendgrid] section where user can specify api_key and > mail_from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AIRFLOW-1723) Support sendgrid in email backend
[
https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222900#comment-16222900
]
Marcin Szymanski edited comment on AIRFLOW-1723 at 10/27/17 9:53 PM:
-
Adding {{**kwargs}} there as well, without any handling inside resolves the
issue. Optionally, there could be warning, that unsupported parameters are
passed.
was (Author: ms32035):
Adding {{**kwargs}} there as well, without any handling inside resolves the
issue.
> Support sendgrid in email backend
> --
>
> Key: AIRFLOW-1723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1723
> Project: Apache Airflow
> Issue Type: New Feature
> Components: utils
>Affects Versions: 1.9.0
>Reporter: Feng Lu
>Assignee: Feng Lu
>Priority: Minor
> Fix For: 1.10.0
>
>
> Current airflow email backend only supports SMTP, this PR aims to extend
> email backend and integrate with sendgrid. The airflow config file is also
> updated to include a [sendgrid] section where user can specify api_key and
> mail_from.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1723) Support sendgrid in email backend
[
https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222900#comment-16222900
]
Marcin Szymanski commented on AIRFLOW-1723:
---
Adding {{**kwargs}} there as well, without any handling inside resolves the
issue.
> Support sendgrid in email backend
> --
>
> Key: AIRFLOW-1723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1723
> Project: Apache Airflow
> Issue Type: New Feature
> Components: utils
>Affects Versions: 1.9.0
>Reporter: Feng Lu
>Assignee: Feng Lu
>Priority: Minor
> Fix For: 1.10.0
>
>
> Current airflow email backend only supports SMTP, this PR aims to extend
> email backend and integrate with sendgrid. The airflow config file is also
> updated to include a [sendgrid] section where user can specify api_key and
> mail_from.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1723) Support sendgrid in email backend
[
https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222865#comment-16222865
]
Marcin Szymanski commented on AIRFLOW-1723:
---
A small request while you're doing this. Can you please add {{**kwargs}} to
{{send_email}} and push it down to the backend. It could be useful for calling
{{Personalization.add_custom_arguments}}, either directly with {{kwargs}} or
with a dictionary from {{kwargs\['custom_arguments'\]}}.
I can prepare prepare a new feature once this is sorted
> Support sendgrid in email backend
> --
>
> Key: AIRFLOW-1723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1723
> Project: Apache Airflow
> Issue Type: New Feature
> Components: utils
>Affects Versions: 1.9.0
>Reporter: Feng Lu
>Assignee: Feng Lu
>Priority: Minor
> Fix For: 1.10.0
>
>
> Current airflow email backend only supports SMTP, this PR aims to extend
> email backend and integrate with sendgrid. The airflow config file is also
> updated to include a [sendgrid] section where user can specify api_key and
> mail_from.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
