[jira] [Created] (AIRFLOW-2636) Airflow crash on click on "Task Duration"
Semet created AIRFLOW-2636:
--
Summary: Airflow crash on click on "Task Duration"
Key: AIRFLOW-2636
URL: https://issues.apache.org/jira/browse/AIRFLOW-2636
Project: Apache Airflow
Issue Type: Bug
Components: webapp
Reporter: Semet
Using Airflow HEAD
{code}
---
Node: afdev-web-55664d57cb-wqxqx
---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in
wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in
full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in
handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in
reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in
full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69,
in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368,
in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_login.py", line 755, in
decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line 270,
in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line
1551, in duration
fails_totals[dict_key] += tf.duration
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2636) Airflow crash on click on "Task Duration"
[
https://issues.apache.org/jira/browse/AIRFLOW-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-2636:
---
Affects Version/s: Airflow 2.0
> Airflow crash on click on "Task Duration"
> -
>
> Key: AIRFLOW-2636
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2636
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: Airflow 2.0
>Reporter: Semet
>Priority: Major
>
> Using Airflow HEAD
> {code}
> ---
> Node: afdev-web-55664d57cb-wqxqx
> ---
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in
> reraise
> raise value
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line
> 368, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/flask_login.py", line 755, in
> decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/www/utils.py", line
> 270, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74,
> in wrapper
> return func(*args, **kwargs)
> File "/usr/local/lib/python3.6/site-packages/airflow/www/views.py", line
> 1551, in duration
> fails_totals[dict_key] += tf.duration
> TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1084) `/health` endpoint on each component
[
https://issues.apache.org/jira/browse/AIRFLOW-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393162#comment-16393162
]
Semet commented on AIRFLOW-1084:
ideally for web, a "ready" end point could also be implemented to drive the
ingress on/off in case of overload. Not really mandatory, but fun to have :)
> `/health` endpoint on each component
>
>
> Key: AIRFLOW-1084
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1084
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Semet
>Priority: Major
>
> Please provide a {{/health}} endpoint of each of the following component:
> - webservice (to avoid pinging the {{/}} root endpoint)
> - worker
> - scheduler
> This would ease integration in Mesos/Marathon framework.
> If you agree, I volunteer to add this change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1084) `/health` endpoint on each component
[
https://issues.apache.org/jira/browse/AIRFLOW-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393063#comment-16393063
]
Semet commented on AIRFLOW-1084:
Hello.
I maintain an Airflow chart for Kubernetes, and most endpoint does not have a
livenessprobe:
- [Scheduler does not have
any|https://github.com/gsemet/kube-airflow/blob/master/airflow/templates/deployments-scheduler.yaml]
- [Web UI have a
/health|https://github.com/gsemet/kube-airflow/blob/master/airflow/templates/deployments-web.yaml]
- [Worker does
not|https://github.com/gsemet/kube-airflow/blob/master/airflow/templates/statefulsets-workers.yaml]
- [Flower does
not|https://github.com/gsemet/kube-airflow/blob/master/airflow/templates/deployments-flower.yaml]
(only relevant for celery executor)
> `/health` endpoint on each component
>
>
> Key: AIRFLOW-1084
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1084
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Semet
>Priority: Major
>
> Please provide a {{/health}} endpoint of each of the following component:
> - webservice (to avoid pinging the {{/}} root endpoint)
> - worker
> - scheduler
> This would ease integration in Mesos/Marathon framework.
> If you agree, I volunteer to add this change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Documenation update
- add new item in the [scheduling
documentation|https://pythonhosted.org/airflow/scheduler.html] about how to
trigger a DAG using a webhook
- describe the sensing dag + processing dag scheme and provide the github use
case as real life example
h2. Possible evolutions
- use an external queue (redis, amqp) to handle lot of events
- subscribe in a pub/sub system such as WAMP?
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- for higher throughput, kafka?
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Documenation update
- add new item in the [scheduling
documentation|https://pythonhosted.org/airflow/scheduler.html] about how to
trigger a DAG using a webhook
- describe the sensing dag + processing dag scheme and provide the github use
case as real life example
h2. Possible evolutions
- use an external queue (redis, amqp) to handle lot of events
- subscribe in a pub/sub ?
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- for higher throughput, kafka?
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Documenation update
- add new item in the [scheduling
documentation|https://pythonhosted.org/airflow/scheduler.html] about how to
trigger a DAG using a webhook
- describe the sensing dag + processing dag scheme and provide the github use
case as real life example
h2. Possible evolutions
- use an external queue (redis, amqp) to handle lot of events
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- for higher throughput, kafka?
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Documenation update
- add new item in the [scheduling
documentation|https://pythonhosted.org/airflow/scheduler.html] about how to
trigger a DAG using a webhook
- describe the sensing dag + processing dag scheme and provide the github use
case as real life example
h2. Possible evolutions
-
- use an external queue (redis, amqp) to handle lot of events
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- for higher throughput, kafka?
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received
[jira] [Commented] (AIRFLOW-1314) Airflow kubernetes integration
[ https://issues.apache.org/jira/browse/AIRFLOW-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265760#comment-16265760 ] Semet commented on AIRFLOW-1314: For information, I am working on an Helm chart for airflow with celery executor: https://github.com/Stibbons/kube-airflow I plan to propose it to the kubernetes chart once I am satisfied with it. Still need to handle dag deployment properly > Airflow kubernetes integration > -- > > Key: AIRFLOW-1314 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1314 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: Airflow 2.0 >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Minor > Labels: features > Fix For: Airflow 2.0 > > > Kubernetes is a container-based cluster management system designed by google > for easy application deployment. Companies such as Airbnb, Bloomberg, > Palantir, and Google use kubernetes for a variety of large-scale solutions > including data science, ETL, and app deployment. Integrating airflow into > Kubernetes would increase viable use cases for airflow, promote airflow as a > de facto workflow scheduler for Kubernetes, and create possibilities for > improved security and robustness within airflow. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-90) `[celery]` section needed even if CeleryExecutor not used
[
https://issues.apache.org/jira/browse/AIRFLOW-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265726#comment-16265726
]
Semet commented on AIRFLOW-90:
--
Please close this issue since it is fixed
> `[celery]` section needed even if CeleryExecutor not used
> -
>
> Key: AIRFLOW-90
> URL: https://issues.apache.org/jira/browse/AIRFLOW-90
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Affects Versions: Airflow 1.7.0
> Environment: * Airflow version: 1.7.0
> * Airflow components: webserver and scheduler with a postgres database and
> LocalExecutor
> * Relevant {{airflow.cfg}} settings: no {{[celery]}} section
> * Python Version: Python 2.7.3
> * Operating System: Linux ubu91 3.13.0-85-generic #129~precise1-Ubuntu SMP
> Fri Mar 18 17:38:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> * Python packages:
> {code}
> Babel==1.3
> Flask==0.10.1
> Flask-Admin==1.4.0
> Flask-Bcrypt==0.7.1
> Flask-Cache==0.13.1
> Flask-Login==0.2.11
> Flask-WTF==0.12
> JPype1==0.6.1
> JayDeBeApi==0.2.0
> Jinja2==2.8
> Mako==1.0.4
> Markdown==2.6.6
> MarkupSafe==0.23
> PyHive==0.1.7
> PySmbClient==0.1.3
> Pygments==2.1.3
> SQLAlchemy==1.0.12
> Sphinx==1.4
> Sphinx-PyPI-upload==0.2.1
> WTForms==2.1
> Werkzeug==0.11.5
> airflow==1.7.0
> alabaster==0.7.7
> alembic==0.8.5
> amqp==1.4.9
> anyjson==0.3.3
> argparse==1.2.1
> backports.ssl-match-hostname==3.5.0.1
> bcrypt==2.0.0
> billiard==3.3.0.23
> boto==2.39.0
> celery==3.1.23
> certifi==2016.2.28
> cffi==1.5.2
> chartkick==0.4.2
> check-manifest==0.31
> coverage==4.0.3
> coveralls==1.1
> croniter==0.3.12
> cryptography==1.3.1
> decorator==4.0.9
> devpi-client==2.5.0
> devpi-common==2.0.8
> dill==0.2.5
> docker-py==1.7.2
> docopt==0.6.2
> docutils==0.12
> enum34==1.1.2
> filechunkio==1.6
> flake8==2.5.4
> flower==0.9.0
> funcsigs==0.4
> future==0.15.2
> futures==3.0.5
> gunicorn==19.3.0
> hdfs==2.0.5
> hive-thrift-py==0.0.1
> idna==2.1
> imagesize==0.7.0
> ipaddress==1.0.16
> ipython==4.1.2
> ipython-genutils==0.1.0
> itsdangerous==0.24
> kombu==3.0.35
> ldap3==1.2.2
> lxml==3.6.0
> mccabe==0.4.0
> mock==1.3.0
> mysqlclient==1.3.7
> nose==1.3.7
> nose-exclude==0.4.1
> numpy==1.11.0
> pandas==0.18.0
> path.py==8.1.2
> pbr==1.8.1
> pep8==1.7.0
> pexpect==4.0.1
> pickleshare==0.6
> pkginfo==1.2.1
> pluggy==0.3.1
> psycopg2==2.6.1
> ptyprocess==0.5.1
> py==1.4.31
> pyOpenSSL==16.0.0
> pyasn1==0.1.9
> pycparser==2.14
> pydruid==0.2.3
> pyflakes==1.0.0
> pykerberos==1.1.10
> pytest==2.9.1
> pytest-cov==2.2.1
> python-dateutil==2.5.2
> python-editor==1.0
> pytz==2016.3
> redis==2.10.5
> requests==2.9.1
> setproctitle==1.1.9
> simplegeneric==0.8.1
> six==1.10.0
> slackclient==1.0.0
> snowballstemmer==1.2.1
> sphinx-argparse==0.1.15
> sphinx-rtd-theme==0.1.9
> statsd==3.2.1
> thrift==0.9.3
> tornado==4.2
> tox==2.3.1
> traitlets==4.2.1
> unicodecsv==0.14.1
> virtualenv==15.0.1
> websocket-client==0.35.0
> wheel==0.29.0
> wsgiref==0.1.2
> {code}
>Reporter: Andreas Merkel
>Priority: Trivial
>
> Now that you know a little about me, let me tell you about the issue I am
> having:
> * What did you expect to happen?
> I expect that if I don't use the Celery Executor, I don't need a {{[celery]}}
> section in my {{airflow.cfg}}.
> * What happened instead?
> If I remove the section, Airflow does not start. At the very least I need
> {code}
> [celery]
> celeryd_concurrency = 1
> {code}
> regardless of the executor configured.
> * Stack trace, if appropriate:
> {code}
> Traceback (most recent call last):
> File "/home/PHI-TPS/amerkel/venv/airflow/bin/airflow", line 13, in
> parser = get_parser()
> File
> "/home/PHI-TPS/amerkel/venv/airflow/local/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 751, in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
> File
> "/home/PHI-TPS/amerkel/venv/airflow/local/lib/python2.7/site-packages/airflow/configuration.py",
> line 520, in get
> return conf.get(section, key, **kwargs)
> File
> "/home/PHI-TPS/amerkel/venv/airflow/local/lib/python2.7/site-packages/airflow/configuration.py",
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key
> [celery/celeryd_concurrency] not found in config
> {code}
> h2. Reproducing the Issue
> Here is how you can reproduce this issue on your machine.
> * Example code that reproduces the issue, including a minimally illustrative
> DAG if necessary:
> {{airflow.cfg}}:
> {code}
> [core]
> airflow_home = $PWD
> dags_folder = $PWD/dags
> base_log_folder = $PWD/airflow_logs
> plugins_folder = $PWD/plugins
> executor = SequentialExecutor
> sql_alchemy_conn = sqlite:///airflow.db
> parallelism = 8
> dag_concurrency = 4
> max_active_runs_per_dag = 4
> load_examples = False
> donot_pickle = False
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Documenation update
- add new item in the [scheduling
documentation|https://pythonhosted.org/airflow/scheduler.html] about how to
trigger a DAG using a webhook
- describe the sensing dag + processing dag scheme and provide the github use
case as real life example
h2. Possible evolutions
-
- use an external queue (redis, amqp) to handle lot of events
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor.
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Possible evolutions:
- use an external queue (redis, amqp) to handle lot of events
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- Security, authentication and other related subject might be adresses in
another ticket.
was:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
h1. Webhook sensor
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
h2. Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
h2. Possible evolutions:
- use an external queue (redis, amqp) to handle lot of events
- allow batch processing (trigger processing DAG on n events or after a
timeout, gathering n messages alltogether)
- Security, authentication and other related subject might be adresses in
another ticket.
was:
h1. Webhook sensor.
May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
To support the charge, I think it is good that the part in the API just post
the received request in an internal queue so the Sensor can handle them later
without risk of missing one.
Documentation would be updated to describe the classic scheme to implement this
use case, which would look like:
!airflow-webhook-proposal.png!
I think it is good to split it into 2 DAGs, one for linear handling of the
messages and triggering new DAG, and the processing DAG that might be executed
in parallel.
Example usage in Sensor DAG: trigger a DAG on GitHub Push Event
{code}
sensor = JsonWebHookSensor(
task_id='my_task_id',
name="on_github_push"
)
.. user is responsible to triggering the processing DAG himself.
{code}
In my github project, I register the following URL in webhook page:
{code}
http://airflow.myserver.com/api/experimental/webhook/on_github_push
{code}
>From now on, on push, github will send a [json with this
>format|https://developer.github.com/v3/activity/events/types/#pushevent] to
>the previous URL.
The {{JsonWebHookSensor}} receives the payload, and a new dag is triggered in
this Sensing Dag.
Security, authentication and other related subject might be adresses in another
ticket.
was:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
!airflow-webhook-proposal.png!
> Webhook Sensor
> --
>
> Key: AIRFLOW-1847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1847
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core, operators
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: api, sensors, webhook
> Attachments: airflow-webhook-proposal.png
>
>
> Webhook sensor. May require a hook in the experimental API
> Register an api endpoint and wait for input on each.
> It is different
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
!airflow-webhook-proposal.png|thumbnail!
was:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
> Webhook Sensor
> --
>
> Key: AIRFLOW-1847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1847
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core, operators
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: api, sensors, webhook
> Attachments: airflow-webhook-proposal.png
>
>
> Webhook sensor. May require a hook in the experimental API
> Register an api endpoint and wait for input on each.
> It is different than the {{dag_runs}} api in that the format is not airflow
> specific, it is just a callback web url called by an external system on some
> even with its application specific content. The content in really important
> and need to be sent to the dag (as XCom?)
> Use Case:
> - A Dag registers a WebHook sensor named {{}}
> - An custom endpoint is exposed at
> {{http://myairflow.server/api/experimental/webhook/}}.
> - I set this URL in the external system I wish to use the webhook from. Ex:
> github/gitlab project webhook
> - when the external application performs a request to this URL, this is
> automatically sent to the WebHook sensor. For simplicity, we can have a
> JsonWebHookSensor that would be able to carry any kind of json content.
> - sensor only job would be normally to trigger the exection of a DAG,
> providing it with the json content as xcom.
> If there are several requests at the same time, the system should be scalable
> enough to not die or not slow down the webui. It is also possible to
> instantiate an independant flask/gunicorn server to split the load. It would
> mean it runs on another port, but this could be just an option in the
> configuration file or even a complete independant application
[jira] [Updated] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1847:
---
Description:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
!airflow-webhook-proposal.png!
was:
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
!airflow-webhook-proposal.png|thumbnail!
> Webhook Sensor
> --
>
> Key: AIRFLOW-1847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1847
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core, operators
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: api, sensors, webhook
> Attachments: airflow-webhook-proposal.png
>
>
> Webhook sensor. May require a hook in the experimental API
> Register an api endpoint and wait for input on each.
> It is different than the {{dag_runs}} api in that the format is not airflow
> specific, it is just a callback web url called by an external system on some
> even with its application specific content. The content in really important
> and need to be sent to the dag (as XCom?)
> Use Case:
> - A Dag registers a WebHook sensor named {{}}
> - An custom endpoint is exposed at
> {{http://myairflow.server/api/experimental/webhook/}}.
> - I set this URL in the external system I wish to use the webhook from. Ex:
> github/gitlab project webhook
> - when the external application performs a request to this URL, this is
> automatically sent to the WebHook sensor. For simplicity, we can have a
> JsonWebHookSensor that would be able to carry any kind of json content.
> - sensor only job would be normally to trigger the exection of a DAG,
> providing it with the json content as xcom.
> If there are several requests at the same time, the system should be scalable
> enough to not die or not slow down the webui. It is also possible to
> instantiate an independant flask/gunicorn server to split the load. It would
> mean it runs on another port, but this could be just an option in the
> configuration file or even a
[jira] [Created] (AIRFLOW-1847) Webhook Sensor
Semet created AIRFLOW-1847:
--
Summary: Webhook Sensor
Key: AIRFLOW-1847
URL: https://issues.apache.org/jira/browse/AIRFLOW-1847
Project: Apache Airflow
Issue Type: Improvement
Components: core, operators
Reporter: Semet
Priority: Minor
Attachments: airflow-webhook-proposal.png
Webhook sensor. May require a hook in the experimental API
Register an api endpoint and wait for input on each.
It is different than the {{dag_runs}} api in that the format is not airflow
specific, it is just a callback web url called by an external system on some
even with its application specific content. The content in really important and
need to be sent to the dag (as XCom?)
Use Case:
- A Dag registers a WebHook sensor named {{}}
- An custom endpoint is exposed at
{{http://myairflow.server/api/experimental/webhook/}}.
- I set this URL in the external system I wish to use the webhook from. Ex:
github/gitlab project webhook
- when the external application performs a request to this URL, this is
automatically sent to the WebHook sensor. For simplicity, we can have a
JsonWebHookSensor that would be able to carry any kind of json content.
- sensor only job would be normally to trigger the exection of a DAG, providing
it with the json content as xcom.
If there are several requests at the same time, the system should be scalable
enough to not die or not slow down the webui. It is also possible to
instantiate an independant flask/gunicorn server to split the load. It would
mean it runs on another port, but this could be just an option in the
configuration file or even a complete independant application ({{airflow
webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
guess it can help this use case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1847) Webhook Sensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet reassigned AIRFLOW-1847:
--
Assignee: Semet
> Webhook Sensor
> --
>
> Key: AIRFLOW-1847
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1847
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core, operators
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: api, sensors, webhook
> Attachments: airflow-webhook-proposal.png
>
>
> Webhook sensor. May require a hook in the experimental API
> Register an api endpoint and wait for input on each.
> It is different than the {{dag_runs}} api in that the format is not airflow
> specific, it is just a callback web url called by an external system on some
> even with its application specific content. The content in really important
> and need to be sent to the dag (as XCom?)
> Use Case:
> - A Dag registers a WebHook sensor named {{}}
> - An custom endpoint is exposed at
> {{http://myairflow.server/api/experimental/webhook/}}.
> - I set this URL in the external system I wish to use the webhook from. Ex:
> github/gitlab project webhook
> - when the external application performs a request to this URL, this is
> automatically sent to the WebHook sensor. For simplicity, we can have a
> JsonWebHookSensor that would be able to carry any kind of json content.
> - sensor only job would be normally to trigger the exection of a DAG,
> providing it with the json content as xcom.
> If there are several requests at the same time, the system should be scalable
> enough to not die or not slow down the webui. It is also possible to
> instantiate an independant flask/gunicorn server to split the load. It would
> mean it runs on another port, but this could be just an option in the
> configuration file or even a complete independant application ({{airflow
> webhookserver}}). I saw recent changes integrated gunicorn in airflow core,
> guess it can help this use case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1755:
---
Description:
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.
I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix = /airflow
...
# The root URL for Flower
flower_url_prefix = /flower
{code}
This will allow deploying airflow on only on a root FQDN, like
{{airflow.mycompany.com}}, but also on a path.
See documentation updated in my commit for example on how to configure reverse
proxy in front of airflow
was:
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.
I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix = /airflow
...
# The root URL for Flower
flower_url_prefix = /flower
{code}
This will allow deploying airflow on only on a root FQDN, like
{{airflow.mycompany.com}}, but also on a path.
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, webapp, webserver
>Affects Versions: 1.10.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: prefix, proxy, reverse, url
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
> See documentation updated in my commit for example on how to configure
> reverse proxy in front of airflow
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1755:
---
Labels: prefix proxy reverse url (was: )
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, webapp, webserver
>Affects Versions: 1.10.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
> Labels: prefix, proxy, reverse, url
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1755:
---
Affects Version/s: 1.10.0
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, webapp, webserver
>Affects Versions: 1.10.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1755:
---
Affects Version/s: (was: Airflow 2.0)
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, webapp, webserver
>Affects Versions: 1.10.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1755:
---
Description:
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.
I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix = /airflow
...
# The root URL for Flower
flower_url_prefix = /flower
{code}
This will allow deploying airflow on only on a root FQDN, like
{{airflow.mycompany.com}}, but also on a path.
was:
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.
I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix: /flower
...
# The root URL for Flower
flower_url_prefix = /flower
{code}
This will allow deploying airflow on only on a root FQDN, like
{{airflow.mycompany.com}}, but also on a path.
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli, core
>Affects Versions: Airflow 2.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix = /airflow
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
[
https://issues.apache.org/jira/browse/AIRFLOW-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218239#comment-16218239
]
Semet commented on AIRFLOW-1755:
Pull Request: https://github.com/apache/incubator-airflow/pull/2723
> URL Prefix for both Flower and Web admin
>
>
> Key: AIRFLOW-1755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli, core
>Affects Versions: Airflow 2.0
>Reporter: Semet
>Assignee: Semet
>Priority: Minor
>
> Similar to AIRFLOW-339.
> Should also fix AIRFLOW-964.
> I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
> arguments:
> {code}
> # Root URL to use for the web server
> web_server_url_prefix: /flower
> ...
> # The root URL for Flower
> flower_url_prefix = /flower
> {code}
> This will allow deploying airflow on only on a root FQDN, like
> {{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AIRFLOW-1755) URL Prefix for both Flower and Web admin
Semet created AIRFLOW-1755:
--
Summary: URL Prefix for both Flower and Web admin
Key: AIRFLOW-1755
URL: https://issues.apache.org/jira/browse/AIRFLOW-1755
Project: Apache Airflow
Issue Type: Improvement
Components: cli, core
Affects Versions: Airflow 2.0
Reporter: Semet
Assignee: Semet
Priority: Minor
Similar to AIRFLOW-339.
Should also fix AIRFLOW-964.
I propose in this change 2 new settings on {{airflow.cfg}} and the CLI
arguments:
{code}
# Root URL to use for the web server
web_server_url_prefix: /flower
...
# The root URL for Flower
flower_url_prefix = /flower
{code}
This will allow deploying airflow on only on a root FQDN, like
{{airflow.mycompany.com}}, but also on a path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AIRFLOW-964) set Flask APPLICATION_ROOT the same to base_url in airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968900#comment-15968900
]
Semet commented on AIRFLOW-964:
---
Look like base_url is not supported by Airflow so far
> set Flask APPLICATION_ROOT the same to base_url in airflow.cfg
> ---
>
> Key: AIRFLOW-964
> URL: https://issues.apache.org/jira/browse/AIRFLOW-964
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: lambdaq
>Priority: Minor
>
> Hello,
> I tried to setup airflow in a restricted host, all web interface has to be
> exposed via a reverse proxy like:
> https://server_ip/fixed_prefix/airflow/
> The reversed proxy is setup properly, airflow homepage was displayed, but
> some of the AJAX API calls failed.
> Open chrome console shows
> /admin/airflow/blocked was 404 not found
> The code was like
> d3.json("/admin/airflow/blocked", function(error, json) {
> $.each(json, function() {
> The template source code was on airflow/www/templates/airflow/list_dags.html
> stroke_width_hover = 6;
> d3.json("{{ url_for('airflow.dag_stats') }}", function(error, json) {
> for(var dag_id in json) {
> states = json[dag_id];
> So I am wondering if we can change Flask's app.config["APPLICATION_ROOT"] to
> the same as
> [webserver]
> baseurl = /fixed_prefix/airflow/
> found in airflow.cfg
> Thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1084) `/health` endpoint on each component
[
https://issues.apache.org/jira/browse/AIRFLOW-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semet updated AIRFLOW-1084:
---
Description:
Please provide a {{/health}} endpoint of each of the following component:
- webservice (to avoid pinging the {{/}} root endpoint)
- worker
- scheduler
This would ease integration in Mesos/Marathon framework.
If you agree, I volunteer to add this change.
was:
Please provide a {{{/health}}} endpoint of each of the following component:
- webservice (to avoid pinging the {{{/}}} root endpoint)
- worker
- scheduler
This would ease integration in Mesos/Marathon framework.
If you agree, I volunteer to add this change.
> `/health` endpoint on each component
>
>
> Key: AIRFLOW-1084
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1084
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Semet
>
> Please provide a {{/health}} endpoint of each of the following component:
> - webservice (to avoid pinging the {{/}} root endpoint)
> - worker
> - scheduler
> This would ease integration in Mesos/Marathon framework.
> If you agree, I volunteer to add this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (AIRFLOW-1084) `/health` endpoint on each component
Semet created AIRFLOW-1084:
--
Summary: `/health` endpoint on each component
Key: AIRFLOW-1084
URL: https://issues.apache.org/jira/browse/AIRFLOW-1084
Project: Apache Airflow
Issue Type: Improvement
Reporter: Semet
Please provide a {{{/health}}} endpoint of each of the following component:
- webservice (to avoid pinging the {{{/}}} root endpoint)
- worker
- scheduler
This would ease integration in Mesos/Marathon framework.
If you agree, I volunteer to add this change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
