[jira] [Updated] (AIRFLOW-2808) Plugin duplication checking is not working
[
https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-2808:
---
Description:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
.
Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working (reason: plugin
method object name is formed using plugin file path + plugin file name + Plugin
Class name. It will never be duplicated given there will not be two files with
the same name in the same directory).
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
was:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
.
Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
> Plugin duplication checking is not working
> --
>
> Key: AIRFLOW-2808
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>
[jira] [Updated] (AIRFLOW-2808) Plugin duplication checking is not working
[
https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-2808:
---
Description:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
.
Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
was:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
. Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
> Plugin duplication checking is not working
> --
>
> Key: AIRFLOW-2808
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> h2. *Background*
> A plugin duplication checking was designed in *plugins_manager.py*
>
[jira] [Updated] (AIRFLOW-2808) Plugin duplication checking is not working
[
https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-2808:
---
Description:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
.
Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
was:
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
.
Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
> Plugin duplication checking is not working
> --
>
> Key: AIRFLOW-2808
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> h2. *Background*
> A plugin duplication checking was designed in *plugins_manager.py*
> [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
[jira] [Created] (AIRFLOW-2808) Plugin duplication checking is not working
Xiaodong DENG created AIRFLOW-2808:
--
Summary: Plugin duplication checking is not working
Key: AIRFLOW-2808
URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
Project: Apache Airflow
Issue Type: Bug
Reporter: Xiaodong DENG
Assignee: Xiaodong DENG
h2. *Background*
A plugin duplication checking was designed in *plugins_manager.py*
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
. Corresponding commit was
[https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
However, it turns out that this checking is not really working.
h2. *Issue*
In my production environment, there are two plugin files with the same name and
operator names in the new _AirflowPlugin_ classes defined inside. However, they
passed the check without any warning or exception.
For example, I have a plugin *file_sensor_1.py* as below,
{code:java}
from airflow.plugins_manager import AirflowPlugin
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
import os
class local_file_sensor(BaseSensorOperator):
@apply_defaults
def __init__(self, file_path, *args, **kwargs):
super(local_file_sensor, self).__init__(*args, **kwargs)
self.file_path = file_path
def poke(self, context):
self.log.info('A-Poking: %s', self.file_path)
return os.path.exists(self.file_path)
class AirflowLocalFileSensorPlugin(AirflowPlugin):
name = "local_file_sensor_plugin"
operators = [local_file_sensor]
{code}
I copy & paste it into another plugin file *file_sensor_2.py*, and make the
only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
me check which one is picked).
Only one plugin would be loaded eventually (because the earlier loaded one will
be overwritten by the later loaded one
[https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
). However, which one? We don't know. It's indeterminate. So far the file name
seems to be the only factor affecting which one would be picked by Airflow.
h2. *My proposal*
Give WARNING to the users when they launch the Airflow. (Or should we give
error msg and fail the launching?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2803) Fix all ESLint issues
[ https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559180#comment-16559180 ] Taylor Edmiston commented on AIRFLOW-2803: -- I took a stab at this one tonight. I started with I think ~1k eslint errors and have resolved ~half of them so far. > Fix all ESLint issues > - > > Key: AIRFLOW-2803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Taylor Edmiston >Priority: Major > > Most of the JS code in Apache Airflow has linting issues which are > highlighted after the integration of ESLint. > Once AIRFLOW-2783 merged in master branch, please fix all the javascript > styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2803) Fix all ESLint issues
[ https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Taylor Edmiston reassigned AIRFLOW-2803: Assignee: Taylor Edmiston > Fix all ESLint issues > - > > Key: AIRFLOW-2803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Taylor Edmiston >Priority: Major > > Most of the JS code in Apache Airflow has linting issues which are > highlighted after the integration of ESLint. > Once AIRFLOW-2783 merged in master branch, please fix all the javascript > styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2805) Display user's local timezone and DAG's timezone on UI
[ https://issues.apache.org/jira/browse/AIRFLOW-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559140#comment-16559140 ] zhangdong edited comment on AIRFLOW-2805 at 7/27/18 2:08 AM: - Great! Displaying local time format in UI is a good idea. But may be a problem, that is crontab expression. Example: What the display will be in timezone Asia/Shanghai about the crontab expression eg:0 0 1 * * was (Author: cctvzd7): Great! Display local time format in UI. > Display user's local timezone and DAG's timezone on UI > -- > > Key: AIRFLOW-2805 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2805 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Priority: Major > > The UI currently only displays the UTC timezone which is also not in human > readable forms on all places. > Make all the date times in human readable forms. > Also, we need to display user's local timezone and DAG's timezone along with > UTC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2805) Display user's local timezone and DAG's timezone on UI
[ https://issues.apache.org/jira/browse/AIRFLOW-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559140#comment-16559140 ] zhangdong commented on AIRFLOW-2805: Great! Display local time format in UI. > Display user's local timezone and DAG's timezone on UI > -- > > Key: AIRFLOW-2805 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2805 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Priority: Major > > The UI currently only displays the UTC timezone which is also not in human > readable forms on all places. > Make all the date times in human readable forms. > Also, we need to display user's local timezone and DAG's timezone along with > UTC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2802) Store dag content and its static conf into DB
[ https://issues.apache.org/jira/browse/AIRFLOW-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangdong reassigned AIRFLOW-2802: -- Assignee: (was: zhangdong) > Store dag content and its static conf into DB > - > > Key: AIRFLOW-2802 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2802 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Reporter: zhangdong >Priority: Major > Labels: newbie > > I am deploying a IDE platform, and I need to add DAG/operate DAG_RUNs va > RestApi。 > Some periodic DAG also has static configs, and these configs should be pass > to the dag_run. > So I propose to store DAG content and its config into DB。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2790) snakebite syntax error: baseTime = min(time * (1L << retries), cap);
[
https://issues.apache.org/jira/browse/AIRFLOW-2790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559093#comment-16559093
]
Yohei Onishi commented on AIRFLOW-2790:
---
Thanks. It seems snakebite depends on Python2 so I will wait the PR gets merged
> snakebite syntax error: baseTime = min(time * (1L << retries), cap);
>
>
> Key: AIRFLOW-2790
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2790
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.9.0
> Environment: Amazon Linux
>Reporter: Yohei Onishi
>Priority: Major
>
> Does anybody know how can I fix this issue?
> * Got the following error when importing
> airflow.operators.sensors.ExternalTaskSensor.
> * apache-airflow 1.9.0 depends on snakebite 2.11.0 and it does not work with
> Python3. https://github.com/spotify/snakebite/issues/250
> [2018-07-23 06:42:51,828] \{models.py:288} ERROR - Failed to import:
> /home/airflow/airflow/dags/example_task_sensor2.py
> Traceback (most recent call last):
> File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 285,
> in process_file
> m = imp.load_source(mod_name, filepath)
> File "/usr/lib64/python3.6/imp.py", line 172, in load_source
> module = _load(spec)
> File "", line 675, in _load
> File "", line 655, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 205, in _call_with_frames_removed
> File "/home/airflow/airflow/dags/example_task_sensor2.py", line 10, in
>
> from airflow.operators.sensors import ExternalTaskSensor
> File "/usr/local/lib/python3.6/site-packages/airflow/operators/sensors.py",
> line 34, in
> from airflow.hooks.hdfs_hook import HDFSHook
> File "/usr/local/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py",
> line 20, in
> from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
> File "/usr/local/lib/python3.6/site-packages/snakebite/client.py", line 1473
> baseTime = min(time * (1L << retries), cap);
> ^
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2807) Add support for External ID when using STS Assume ROle
Vojtech Vondra created AIRFLOW-2807: --- Summary: Add support for External ID when using STS Assume ROle Key: AIRFLOW-2807 URL: https://issues.apache.org/jira/browse/AIRFLOW-2807 Project: Apache Airflow Issue Type: Improvement Components: aws, boto3, hooks Affects Versions: 1.10.1 Reporter: Vojtech Vondra Currently the role assumption method works only if the granting account does not specify an External ID. The external ID is used to solved the confused deputy problem. When using the AWS hook to export data to multiple customers, it's good security practice to use the external ID. Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2766) Replicated *Base date* in each tab on DAG detail view
[ https://issues.apache.org/jira/browse/AIRFLOW-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-2766. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3615 [https://github.com/apache/incubator-airflow/pull/3615] > Replicated *Base date* in each tab on DAG detail view > - > > Key: AIRFLOW-2766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2766 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-07-18 at 1.57.22 PM.png > > > At the moment, most of the tabs on DAG detail page has the *Base date* > feature which behaves only for that particular tab/view. From end-user's > perspective it is confusing as a user assume to select a date and then see > all views based on that date > The idea is to move the *Base date* outside the tabs, and each tab will then > behave on the *Base date* on the Global level. Users will then only need to > select the base date once and every tab will behave on that date. > Please see attached screenshot for reference > Or, the second approach could be to make sure the change of Base Date in each > tab remains the same and doesn't change on page refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2766) Replicated *Base date* in each tab on DAG detail view
[ https://issues.apache.org/jira/browse/AIRFLOW-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558769#comment-16558769 ] ASF subversion and git services commented on AIRFLOW-2766: -- Commit 25f31c9b773b1bf61738282c8e4ab997fce57e45 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=25f31c9 ] [AIRFLOW-2766][Airflow-2766] Respect shared datetime across tabs Closes #3615 from verdan/AIRFLOW-2766-shared- datetime > Replicated *Base date* in each tab on DAG detail view > - > > Key: AIRFLOW-2766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2766 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-07-18 at 1.57.22 PM.png > > > At the moment, most of the tabs on DAG detail page has the *Base date* > feature which behaves only for that particular tab/view. From end-user's > perspective it is confusing as a user assume to select a date and then see > all views based on that date > The idea is to move the *Base date* outside the tabs, and each tab will then > behave on the *Base date* on the Global level. Users will then only need to > select the base date once and every tab will behave on that date. > Please see attached screenshot for reference > Or, the second approach could be to make sure the change of Base Date in each > tab remains the same and doesn't change on page refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2766) Replicated *Base date* in each tab on DAG detail view
[ https://issues.apache.org/jira/browse/AIRFLOW-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558768#comment-16558768 ] ASF subversion and git services commented on AIRFLOW-2766: -- Commit 25f31c9b773b1bf61738282c8e4ab997fce57e45 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=25f31c9 ] [AIRFLOW-2766][Airflow-2766] Respect shared datetime across tabs Closes #3615 from verdan/AIRFLOW-2766-shared- datetime > Replicated *Base date* in each tab on DAG detail view > - > > Key: AIRFLOW-2766 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2766 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Verdan Mahmood >Priority: Minor > Fix For: 2.0.0 > > Attachments: Screen Shot 2018-07-18 at 1.57.22 PM.png > > > At the moment, most of the tabs on DAG detail page has the *Base date* > feature which behaves only for that particular tab/view. From end-user's > perspective it is confusing as a user assume to select a date and then see > all views based on that date > The idea is to move the *Base date* outside the tabs, and each tab will then > behave on the *Base date* on the Global level. Users will then only need to > select the base date once and every tab will behave on that date. > Please see attached screenshot for reference > Or, the second approach could be to make sure the change of Base Date in each > tab remains the same and doesn't change on page refresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-2776. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3620 [https://github.com/apache/incubator-airflow/pull/3620] > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2776) Tree view JSON is unnecessarily large
[ https://issues.apache.org/jira/browse/AIRFLOW-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558765#comment-16558765 ] ASF subversion and git services commented on AIRFLOW-2776: -- Commit 68535d1886f22b0fa833727aac4d0ee3ce12edf1 in incubator-airflow's branch refs/heads/master from [~abdul-stripe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=68535d1 ] [AIRFLOW-2776] Compress tree view JSON The tree view generates JSON that can be massive for bigger DAGs, up to 10s of MBs. The JSON is currently prettified, which both takes up more CPU time during serialization, and slows down everything else that uses it. Considering the JSON is only meant to be used programmatically, this is an easy win Closes #3620 from abdul-stripe/smaller-tree-view- json > Tree view JSON is unnecessarily large > - > > Key: AIRFLOW-2776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2776 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > The tree view generates JSON that can be massive for bigger DAGs –– some of > our tree views at stripe have 10s of MBs of JSON. > The [generated JSON is > prettified|https://github.com/apache/incubator-airflow/blob/52c745da71a6da798f7322956967b5e818b56e48/airflow/www/views.py#L1480], > which both takes up more CPU time during serialization, as well as slowing > everything else that uses it. We patched this on stripe's fork by removing > all whitespace and had a ton of speedup for bigger tree views. Considering > the JSON is only meant to be used programmatically, this is probably an easy > win. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2777) dag.sub_dag(...) speedups
[ https://issues.apache.org/jira/browse/AIRFLOW-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558761#comment-16558761 ] ASF subversion and git services commented on AIRFLOW-2777: -- Commit 4cd72b91bd024e6be9abb30aee345c0a90a6561b in incubator-airflow's branch refs/heads/master from [~abdul-stripe] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4cd72b9 ] [AIRFLOW-2777] speed up dag.sub_dag(...) previous version created the subdag by copying over all the tasks, and then filtering them down. it's a lot faster if we only copy over the tasks we need Closes #3621 from abdul-stripe/faster-subdag > dag.sub_dag(...) speedups > - > > Key: AIRFLOW-2777 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2777 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > dag.sub_dag(...) currently works by first deep copying the entire dag, and > then filtering down to the appropriate tasks. that can be slow since deep > copying a big dag takes a while. specifically, copying over all the tasks is > the bottleneck > this can be a lot faster by instead only copying over the filtered down tasks > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2776] Compress tree view JSON
Repository: incubator-airflow
Updated Branches:
refs/heads/master 4cd72b91b -> 68535d188
[AIRFLOW-2776] Compress tree view JSON
The tree view generates JSON that can be massive
for bigger DAGs,
up to 10s of MBs. The JSON is currently
prettified, which both
takes up more CPU time during serialization, and
slows down
everything else that uses it. Considering the JSON
is only
meant to be used programmatically, this is an easy
win
Closes #3620 from abdul-stripe/smaller-tree-view-
json
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/68535d18
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/68535d18
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/68535d18
Branch: refs/heads/master
Commit: 68535d1886f22b0fa833727aac4d0ee3ce12edf1
Parents: 4cd72b9
Author: Abdul Nimeri
Authored: Thu Jul 26 20:53:57 2018 +0200
Committer: Bolke de Bruin
Committed: Thu Jul 26 20:53:57 2018 +0200
--
airflow/www/views.py | 3 ++-
airflow/www_rbac/views.py | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/68535d18/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index ddd2765..178a8e6 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -1477,7 +1477,8 @@ class Airflow(BaseView):
for d in dates],
}
-data = json.dumps(data, indent=4, default=json_ser)
+# minimize whitespace as this can be huge for bigger dags
+data = json.dumps(data, default=json_ser, separators=(',', ':'))
session.commit()
form = DateTimeWithNumRunsForm(data={'base_date': max_date,
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/68535d18/airflow/www_rbac/views.py
--
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 44efac9..4673def 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -1208,7 +1208,8 @@ class Airflow(AirflowBaseView):
for d in dates],
}
-data = json.dumps(data, indent=4, default=json_ser)
+# minimize whitespace as this can be huge for bigger dags
+data = json.dumps(data, default=json_ser, separators=(',', ':'))
session.commit()
form = DateTimeWithNumRunsForm(data={'base_date': max_date,
[jira] [Resolved] (AIRFLOW-2777) dag.sub_dag(...) speedups
[ https://issues.apache.org/jira/browse/AIRFLOW-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-2777. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3621 [https://github.com/apache/incubator-airflow/pull/3621] > dag.sub_dag(...) speedups > - > > Key: AIRFLOW-2777 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2777 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Abdul Nimeri >Assignee: Abdul Nimeri >Priority: Minor > Fix For: 2.0.0 > > > dag.sub_dag(...) currently works by first deep copying the entire dag, and > then filtering down to the appropriate tasks. that can be slow since deep > copying a big dag takes a while. specifically, copying over all the tasks is > the bottleneck > this can be a lot faster by instead only copying over the filtered down tasks > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2777] speed up dag.sub_dag(...)
Repository: incubator-airflow
Updated Branches:
refs/heads/master 8c441a0d8 -> 4cd72b91b
[AIRFLOW-2777] speed up dag.sub_dag(...)
previous version created the subdag by copying
over all the tasks, and
then filtering them down. it's a lot faster if we
only copy over the
tasks we need
Closes #3621 from abdul-stripe/faster-subdag
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/4cd72b91
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/4cd72b91
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/4cd72b91
Branch: refs/heads/master
Commit: 4cd72b91bd024e6be9abb30aee345c0a90a6561b
Parents: 8c441a0
Author: Abdul Nimeri
Authored: Thu Jul 26 20:53:11 2018 +0200
Committer: Bolke de Bruin
Committed: Thu Jul 26 20:53:11 2018 +0200
--
airflow/models.py | 11 +--
1 file changed, 9 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4cd72b91/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index d6a9199..1d832ab 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -3905,10 +3905,15 @@ class DAG(BaseDag, LoggingMixin):
upstream and downstream neighbours based on the flag passed.
"""
+# deep-copying self.task_dict takes a long time, and we don't want all
+# the tasks anyway, so we copy the tasks manually later
+task_dict = self.task_dict
+self.task_dict = {}
dag = copy.deepcopy(self)
+self.task_dict = task_dict
regex_match = [
-t for t in dag.tasks if re.findall(task_regex, t.task_id)]
+t for t in self.tasks if re.findall(task_regex, t.task_id)]
also_include = []
for t in regex_match:
if include_downstream:
@@ -3917,7 +3922,9 @@ class DAG(BaseDag, LoggingMixin):
also_include += t.get_flat_relatives(upstream=True)
# Compiling the unique list of tasks that made the cut
-dag.task_dict = {t.task_id: t for t in regex_match + also_include}
+# Make sure to not recursively deepcopy the dag while copying the task
+dag.task_dict = {t.task_id: copy.deepcopy(t, {id(t.dag): t.dag})
+ for t in regex_match + also_include}
for t in dag.tasks:
# Removing upstream/downstream references to tasks that did not
# made the cut
[jira] [Commented] (AIRFLOW-2765) Set default mime_charset in email_operator.EmailOperator to UTF-8, to match email utils
[
https://issues.apache.org/jira/browse/AIRFLOW-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558755#comment-16558755
]
ASF subversion and git services commented on AIRFLOW-2765:
--
Commit 8c441a0d88627fb97b465c5045bdcad24f1d305c in incubator-airflow's branch
refs/heads/master from Jeffrey Scott Keone Payne
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8c441a0 ]
[AIRFLOW-2765] Set default mime_charset to UTF-8
Closes #3627 from jeffkpayne/AIRFLOW-2765
> Set default mime_charset in email_operator.EmailOperator to UTF-8, to match
> email utils
> ---
>
> Key: AIRFLOW-2765
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2765
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Major
> Fix For: 2.0.0
>
>
> Assuming we would want both {{EmailOperator()}} and
> {{utils.email.send_email()}} to use the same default value for the
> {{mime_charset}} parameter. The default for {{utils.email.send_email()}} was
> recently changed to {{UTF-8}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2765) Set default mime_charset in email_operator.EmailOperator to UTF-8, to match email utils
[
https://issues.apache.org/jira/browse/AIRFLOW-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558754#comment-16558754
]
ASF subversion and git services commented on AIRFLOW-2765:
--
Commit 8c441a0d88627fb97b465c5045bdcad24f1d305c in incubator-airflow's branch
refs/heads/master from Jeffrey Scott Keone Payne
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8c441a0 ]
[AIRFLOW-2765] Set default mime_charset to UTF-8
Closes #3627 from jeffkpayne/AIRFLOW-2765
> Set default mime_charset in email_operator.EmailOperator to UTF-8, to match
> email utils
> ---
>
> Key: AIRFLOW-2765
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2765
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Major
> Fix For: 2.0.0
>
>
> Assuming we would want both {{EmailOperator()}} and
> {{utils.email.send_email()}} to use the same default value for the
> {{mime_charset}} parameter. The default for {{utils.email.send_email()}} was
> recently changed to {{UTF-8}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2765] Set default mime_charset to UTF-8
Repository: incubator-airflow Updated Branches: refs/heads/master 604ea1dcc -> 8c441a0d8 [AIRFLOW-2765] Set default mime_charset to UTF-8 Closes #3627 from jeffkpayne/AIRFLOW-2765 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8c441a0d Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8c441a0d Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8c441a0d Branch: refs/heads/master Commit: 8c441a0d88627fb97b465c5045bdcad24f1d305c Parents: 604ea1d Author: Jeffrey Scott Keone Payne Authored: Thu Jul 26 20:52:21 2018 +0200 Committer: Bolke de Bruin Committed: Thu Jul 26 20:52:21 2018 +0200 -- airflow/operators/email_operator.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8c441a0d/airflow/operators/email_operator.py -- diff --git a/airflow/operators/email_operator.py b/airflow/operators/email_operator.py index ae176ca..7b4109c 100644 --- a/airflow/operators/email_operator.py +++ b/airflow/operators/email_operator.py @@ -60,7 +60,7 @@ class EmailOperator(BaseOperator): cc=None, bcc=None, mime_subtype='mixed', -mime_charset='us_ascii', +mime_charset='utf-8', *args, **kwargs): super(EmailOperator, self).__init__(*args, **kwargs) self.to = to
[jira] [Resolved] (AIRFLOW-2765) Set default mime_charset in email_operator.EmailOperator to UTF-8, to match email utils
[
https://issues.apache.org/jira/browse/AIRFLOW-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin resolved AIRFLOW-2765.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request #3627
[https://github.com/apache/incubator-airflow/pull/3627]
> Set default mime_charset in email_operator.EmailOperator to UTF-8, to match
> email utils
> ---
>
> Key: AIRFLOW-2765
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2765
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Major
> Fix For: 2.0.0
>
>
> Assuming we would want both {{EmailOperator()}} and
> {{utils.email.send_email()}} to use the same default value for the
> {{mime_charset}} parameter. The default for {{utils.email.send_email()}} was
> recently changed to {{UTF-8}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2783) Integrate `eslint` with a decent set of rules
[ https://issues.apache.org/jira/browse/AIRFLOW-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin closed AIRFLOW-2783. --- Resolution: Fixed Fix Version/s: 2.0.0 > Integrate `eslint` with a decent set of rules > - > > Key: AIRFLOW-2783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Priority: Major > Fix For: 2.0.0 > > > To make sure that the javascript code is standardized and well written, we > should integrate eslint with a decent set of rules, so that everyone follows > those rules while writing the JS. > https://eslint.org/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2783) Integrate `eslint` with a decent set of rules
[ https://issues.apache.org/jira/browse/AIRFLOW-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558742#comment-16558742 ] ASF subversion and git services commented on AIRFLOW-2783: -- Commit 604ea1dccc256d10a03547e7d330f5d3ce1389f8 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=604ea1d ] [AIRFLOW-2783][Airflow 2783] Implement eslint for JS code check Closes #3641 from verdan/AIRFLOW-2783-eslint > Integrate `eslint` with a decent set of rules > - > > Key: AIRFLOW-2783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Priority: Major > > To make sure that the javascript code is standardized and well written, we > should integrate eslint with a decent set of rules, so that everyone follows > those rules while writing the JS. > https://eslint.org/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2783) Integrate `eslint` with a decent set of rules
[ https://issues.apache.org/jira/browse/AIRFLOW-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558741#comment-16558741 ] ASF subversion and git services commented on AIRFLOW-2783: -- Commit 604ea1dccc256d10a03547e7d330f5d3ce1389f8 in incubator-airflow's branch refs/heads/master from [~verdan] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=604ea1d ] [AIRFLOW-2783][Airflow 2783] Implement eslint for JS code check Closes #3641 from verdan/AIRFLOW-2783-eslint > Integrate `eslint` with a decent set of rules > - > > Key: AIRFLOW-2783 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2783 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Priority: Major > > To make sure that the javascript code is standardized and well written, we > should integrate eslint with a decent set of rules, so that everyone follows > those rules while writing the JS. > https://eslint.org/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2783][Airflow 2783] Implement eslint for JS code check
Repository: incubator-airflow
Updated Branches:
refs/heads/master 10b6ca5e4 -> 604ea1dcc
[AIRFLOW-2783][Airflow 2783] Implement eslint for JS code check
Closes #3641 from verdan/AIRFLOW-2783-eslint
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/604ea1dc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/604ea1dc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/604ea1dc
Branch: refs/heads/master
Commit: 604ea1dccc256d10a03547e7d330f5d3ce1389f8
Parents: 10b6ca5
Author: Verdan Mahmood
Authored: Thu Jul 26 20:45:14 2018 +0200
Committer: Bolke de Bruin
Committed: Thu Jul 26 20:45:14 2018 +0200
--
.rat-excludes |2 +
CONTRIBUTING.md| 16 +
airflow/www_rbac/.eslintignore |8 +
airflow/www_rbac/.eslintrc |5 +
airflow/www_rbac/package-lock.json | 1269 ++-
airflow/www_rbac/package.json | 12 +-
airflow/www_rbac/webpack.config.js | 44 +-
7 files changed, 1326 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/604ea1dc/.rat-excludes
--
diff --git a/.rat-excludes b/.rat-excludes
index 0164f99..b13c164 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -3,6 +3,8 @@
.coverage
.coveragerc
.codecov.yml
+.eslintrc
+.eslintignore
.rat-excludes
requirements.txt
.*log
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/604ea1dc/CONTRIBUTING.md
--
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 94997ae..47a1a80 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -299,3 +299,19 @@ npm run dev
Should you add or upgrade a npm package, which involves changing
`package.json`, you'll need to re-run `npm install`
and push the newly generated `package-lock.json` file so we get the
reproducible build.
+ Javascript Style Guide
+
+We try to enforce a more consistent style and try to follow the JS community
guidelines.
+Once you add or modify any javascript code in the project, please make sure it
follows the guidelines
+defined in [Airbnb JavaScript Style
Guide](https://github.com/airbnb/javascript).
+Apache Airflow uses [ESLint](https://eslint.org/) as a tool for identifying
and reporting on patterns in JavaScript,
+which can be used by running any of the following commands.
+
+```bash
+# Check JS code in .js and .html files, and report any errors/warnings
+npm run lint
+
+# Check JS code in .js and .html files, report any errors/warnings and fix
them if possible
+npm run lint:fix
+```
+
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/604ea1dc/airflow/www_rbac/.eslintignore
--
diff --git a/airflow/www_rbac/.eslintignore b/airflow/www_rbac/.eslintignore
new file mode 100644
index 000..924c9fc
--- /dev/null
+++ b/airflow/www_rbac/.eslintignore
@@ -0,0 +1,8 @@
+**/*{.,-}min.js
+**/*.sh
+**/*.py
+gantt-chart-d3v2.js
+jqClock.min.js
+coverage/**
+static/dist/*
+static/docs/*
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/604ea1dc/airflow/www_rbac/.eslintrc
--
diff --git a/airflow/www_rbac/.eslintrc b/airflow/www_rbac/.eslintrc
new file mode 100644
index 000..e566b6e
--- /dev/null
+++ b/airflow/www_rbac/.eslintrc
@@ -0,0 +1,5 @@
+{
+ "extends": "airbnb-base",
+ "parser": "babel-eslint",
+ "plugins": [ "html" ]
+}
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/604ea1dc/airflow/www_rbac/package-lock.json
--
diff --git a/airflow/www_rbac/package-lock.json
b/airflow/www_rbac/package-lock.json
index a5cbf24..7038c4e 100644
--- a/airflow/www_rbac/package-lock.json
+++ b/airflow/www_rbac/package-lock.json
@@ -2,6 +2,193 @@
"requires": true,
"lockfileVersion": 1,
"dependencies": {
+"@babel/code-frame": {
+ "version": "7.0.0-beta.44",
+ "resolved":
"https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.0.0-beta.44.tgz;,
+ "integrity":
"sha512-cuAuTTIQ9RqcFRJ/Y8PvTh+paepNcaGxwQwjIDRWPXmzzyAeCO4KqS9ikMvq0MCbRk6GlYKwfzStrcP3/jSL8g==",
+ "dev": true,
+ "requires": {
+"@babel/highlight": "7.0.0-beta.44"
+ }
+},
+"@babel/generator": {
+ "version": "7.0.0-beta.44",
+ "resolved":
"https://registry.npmjs.org/@babel/generator/-/generator-7.0.0-beta.44.tgz;,
+ "integrity":
"sha512-5xVb7hlhjGcdkKpMXgicAVgx8syK5VJz193k0i/0sLP6DzE6lRrU1K3B/rFefgdo9LPGMAOOOAWW4jycj07ShQ==",
+ "dev": true,
+ "requires": {
+
[jira] [Updated] (AIRFLOW-2806) test_mark_success_no_kill test breaks intermittently on CI
[
https://issues.apache.org/jira/browse/AIRFLOW-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taylor Edmiston updated AIRFLOW-2806:
-
Description:
The test_mark_success_no_kill test is breaking intermittently on the CI for
some versions of Python and some databases, particularly Python 3.5 for both
PostgreSQL and MySQL.
A traceback of the error is
([link|https://travis-ci.org/apache/incubator-airflow/jobs/407522994#L5668-L5701]):
{code:java}
10) ERROR: test_mark_success_no_kill (tests.transplant_class..C)
--
Traceback (most recent call last):
tests/jobs.py line 1116 in test_mark_success_no_kill
ti.refresh_from_db()
airflow/utils/db.py line 74 in wrapper
return func(*args, **kwargs)
/opt/python/3.5.5/lib/python3.5/contextlib.py line 66 in __exit__
next(self.gen)
airflow/utils/db.py line 45 in create_session
session.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 927 in commit
self.transaction.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 471 in commit
t[1].commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1632 in commit
self._do_commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1663 in _do_commit
self.connection._commit_impl()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 723 in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1402 in _handle_dbapi_exception
exc_info
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/util/compat.py
line 203 in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/util/compat.py
line 186 in reraise
raise value.with_traceback(tb)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 721 in _commit_impl
self.engine.dialect.do_commit(self.connection)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/default.py
line 443 in do_commit
dbapi_connection.commit()
OperationalError: (psycopg2.OperationalError) server closed the connection
unexpectedly
This probably means the server terminated abnormally{code}
It seems to be erroring out on trying to
[commit|http://initd.org/psycopg/docs/connection.html#connection.commit] the
pending transaction to the database, possibly because the connection has been
closed. What's weird is that this line is already in a try-except block
catching all exceptions, but I think it's somehow not entering the except
clause.
[https://github.com/apache/incubator-airflow/blob/f3b6b60c4809afdde916e8982a300f942f26109b/airflow/utils/db.py#L36-L50]
Note: This is a follow up to AIRFLOW-2801 ([PR
#3642|https://github.com/apache/incubator-airflow/pull/3642]) which provided a
short-term solution by skipping the flaky test.
was:
The test_mark_success_no_kill test is breaking intermittently on the CI for
some versions of Python and some databases, particularly Python 3.5 for both
PostgreSQL and MySQL.
A traceback of the error is
([link|https://travis-ci.org/apache/incubator-airflow/jobs/407522994#L5668-L5701]):
{code:java}
10) ERROR: test_mark_success_no_kill (tests.transplant_class..C)
--
Traceback (most recent call last):
tests/jobs.py line 1116 in test_mark_success_no_kill
ti.refresh_from_db()
airflow/utils/db.py line 74 in wrapper
return func(*args, **kwargs)
/opt/python/3.5.5/lib/python3.5/contextlib.py line 66 in __exit__
next(self.gen)
airflow/utils/db.py line 45 in create_session
session.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 927 in commit
self.transaction.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 471 in commit
t[1].commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1632 in commit
self._do_commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1663 in _do_commit
self.connection._commit_impl()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 723 in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1402 in _handle_dbapi_exception
exc_info
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/util/compat.py
line 203 in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
[jira] [Created] (AIRFLOW-2806) test_mark_success_no_kill test breaks intermittently on CI
Taylor Edmiston created AIRFLOW-2806:
Summary: test_mark_success_no_kill test breaks intermittently on CI
Key: AIRFLOW-2806
URL: https://issues.apache.org/jira/browse/AIRFLOW-2806
Project: Apache Airflow
Issue Type: Bug
Reporter: Taylor Edmiston
Assignee: Taylor Edmiston
The test_mark_success_no_kill test is breaking intermittently on the CI for
some versions of Python and some databases, particularly Python 3.5 for both
PostgreSQL and MySQL.
A traceback of the error is
([link|https://travis-ci.org/apache/incubator-airflow/jobs/407522994#L5668-L5701]):
{code:java}
10) ERROR: test_mark_success_no_kill (tests.transplant_class..C)
--
Traceback (most recent call last):
tests/jobs.py line 1116 in test_mark_success_no_kill
ti.refresh_from_db()
airflow/utils/db.py line 74 in wrapper
return func(*args, **kwargs)
/opt/python/3.5.5/lib/python3.5/contextlib.py line 66 in __exit__
next(self.gen)
airflow/utils/db.py line 45 in create_session
session.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 927 in commit
self.transaction.commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/orm/session.py
line 471 in commit
t[1].commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1632 in commit
self._do_commit()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1663 in _do_commit
self.connection._commit_impl()
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 723 in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 1402 in _handle_dbapi_exception
exc_info
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/util/compat.py
line 203 in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/util/compat.py
line 186 in reraise
raise value.with_traceback(tb)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/base.py
line 721 in _commit_impl
self.engine.dialect.do_commit(self.connection)
.tox/py35-backend_postgres/lib/python3.5/site-packages/sqlalchemy/engine/default.py
line 443 in do_commit
dbapi_connection.commit()
OperationalError: (psycopg2.OperationalError) server closed the connection
unexpectedly
This probably means the server terminated abnormally{code}
It seems to be erroring out on trying to
[commit|http://initd.org/psycopg/docs/connection.html#connection.commit] the
pending transaction to the database, possibly because the connection has been
closed. What's weird is that this line is already in a try-except block
catching all exceptions, but I think it's somehow not entering the except
clause.
https://github.com/apache/incubator-airflow/blob/f3b6b60c4809afdde916e8982a300f942f26109b/airflow/utils/db.py#L36-L50
Note: This is a follow up to AIRFLOW-2801 (Jira issue #2801 [1], GitHub PR
#3642 [2]) which provided a short-term solution by skipping the flaky test.
[1]: https://issues.apache.org/jira/browse/AIRFLOW-2801
[2]: [https://github.com/apache/incubator-airflow/pull/3642]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2805) Display user's local timezone and DAG's timezone on UI
Verdan Mahmood created AIRFLOW-2805: --- Summary: Display user's local timezone and DAG's timezone on UI Key: AIRFLOW-2805 URL: https://issues.apache.org/jira/browse/AIRFLOW-2805 Project: Apache Airflow Issue Type: Improvement Reporter: Verdan Mahmood The UI currently only displays the UTC timezone which is also not in human readable forms on all places. Make all the date times in human readable forms. Also, we need to display user's local timezone and DAG's timezone along with UTC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2730) Airflow 1.9.0+ Web UI Fails to Load in IE11
[ https://issues.apache.org/jira/browse/AIRFLOW-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558150#comment-16558150 ] Verdan Mahmood commented on AIRFLOW-2730: - I believe it would make more sense to pick this issue after extracting all inline JS from html files into separate .js file. I've created a new JIRA issue to do that. https://issues.apache.org/jira/browse/AIRFLOW-2804 > Airflow 1.9.0+ Web UI Fails to Load in IE11 > --- > > Key: AIRFLOW-2730 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2730 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.9.0 >Reporter: Cameron Yick >Priority: Minor > > As a developer, I would like to use Airflow in enterprise environments where > IE11 is the only browser available. > Presently, the admin view doesn't load because some of the inlined javascript > on the page uses ES6 features, like the array spread operator. > Fixing this change will become a lot easier after AIRFLOW-2691 goes through, > because transpilation could happen as part of the build process. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2804) Extract inline JS from html files and move them in separate .js files
Verdan Mahmood created AIRFLOW-2804: --- Summary: Extract inline JS from html files and move them in separate .js files Key: AIRFLOW-2804 URL: https://issues.apache.org/jira/browse/AIRFLOW-2804 Project: Apache Airflow Issue Type: Improvement Reporter: Verdan Mahmood Apache Airflow uses inline JS code on most/all of the pages, making it difficult to optimize/minimfy them properly. Please make full use of npm and webpack by extracting all inline JS from html files, moving them in separate .js files and make use of `require` and `import` to optimize the dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2803) Fix all ESLint issues
Verdan Mahmood created AIRFLOW-2803: --- Summary: Fix all ESLint issues Key: AIRFLOW-2803 URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 Project: Apache Airflow Issue Type: Improvement Reporter: Verdan Mahmood Most of the JS code in Apache Airflow has linting issues which are highlighted after the integration of ESLint. Once AIRFLOW-2783 merged in master branch, please fix all the javascript styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2801) test_mark_success_no_kill test is flaky on CI for Postgres
[ https://issues.apache.org/jira/browse/AIRFLOW-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2801. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3642 [https://github.com/apache/incubator-airflow/pull/3642] > test_mark_success_no_kill test is flaky on CI for Postgres > -- > > Key: AIRFLOW-2801 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2801 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > Fix For: 2.0.0 > > > Per the mailing list thread "Flaky test case: test_mark_success_no_kill", > this test is failing regularly on the CI in PostgreSQL. We already skip it > for MySQL, so this just adds a similar skip for Postgres. > Example - https://travis-ci.org/apache/incubator-airflow/jobs/407522994 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2801) test_mark_success_no_kill test is flaky on CI for Postgres
[ https://issues.apache.org/jira/browse/AIRFLOW-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558082#comment-16558082 ] ASF subversion and git services commented on AIRFLOW-2801: -- Commit 10b6ca5e4808c0ac948456e1004f9f64aae54a4d in incubator-airflow's branch refs/heads/master from Taylor D. Edmiston [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=10b6ca5 ] [AIRFLOW-2801] Skip test_mark_success_no_kill in PostgreSQL on CI See mailing list thread "Flaky test case: test_mark_success_no_kill". Closes #3642 from tedmiston/test_mark_success_no_kill-postgresql > test_mark_success_no_kill test is flaky on CI for Postgres > -- > > Key: AIRFLOW-2801 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2801 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Reporter: Taylor Edmiston >Assignee: Taylor Edmiston >Priority: Minor > > Per the mailing list thread "Flaky test case: test_mark_success_no_kill", > this test is failing regularly on the CI in PostgreSQL. We already skip it > for MySQL, so this just adds a similar skip for Postgres. > Example - https://travis-ci.org/apache/incubator-airflow/jobs/407522994 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2801] Skip test_mark_success_no_kill in PostgreSQL on CI
Repository: incubator-airflow
Updated Branches:
refs/heads/master c06e5fa1e -> 10b6ca5e4
[AIRFLOW-2801] Skip test_mark_success_no_kill in PostgreSQL on CI
See mailing list thread "Flaky test case:
test_mark_success_no_kill".
Closes #3642 from
tedmiston/test_mark_success_no_kill-postgresql
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/10b6ca5e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/10b6ca5e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/10b6ca5e
Branch: refs/heads/master
Commit: 10b6ca5e4808c0ac948456e1004f9f64aae54a4d
Parents: c06e5fa
Author: Taylor D. Edmiston
Authored: Thu Jul 26 11:02:47 2018 +0200
Committer: Fokko Driesprong
Committed: Thu Jul 26 11:02:47 2018 +0200
--
tests/jobs.py | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/10b6ca5e/tests/jobs.py
--
diff --git a/tests/jobs.py b/tests/jobs.py
index cda4517..93f6574 100644
--- a/tests/jobs.py
+++ b/tests/jobs.py
@@ -1088,6 +1088,8 @@ class LocalTaskJobTest(unittest.TestCase):
@unittest.skipIf('mysql' in configuration.conf.get('core',
'sql_alchemy_conn'),
"flaky when run on mysql")
+@unittest.skipIf('postgresql' in configuration.conf.get('core',
'sql_alchemy_conn'),
+ 'flaky when run on postgresql')
def test_mark_success_no_kill(self):
"""
Test that ensures that mark_success in the UI doesn't cause
[jira] [Resolved] (AIRFLOW-2791) Add Hipages to list of current airflow users
[ https://issues.apache.org/jira/browse/AIRFLOW-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AIRFLOW-2791. --- Resolution: Fixed Issue resolved by pull request #3632 [https://github.com/apache/incubator-airflow/pull/3632] > Add Hipages to list of current airflow users > > > Key: AIRFLOW-2791 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2791 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Arihant >Assignee: Arihant >Priority: Trivial > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2791] Add Hipages to list of current airflow users
Repository: incubator-airflow Updated Branches: refs/heads/master 8715bc45b -> c06e5fa1e [AIRFLOW-2791] Add Hipages to list of current airflow users Closes #3632 from arihantsurana/AIRFLOW-2791 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c06e5fa1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c06e5fa1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c06e5fa1 Branch: refs/heads/master Commit: c06e5fa1ecff731e0a20026dcee91e11061008be Parents: 8715bc4 Author: arihantsurana Authored: Thu Jul 26 10:50:39 2018 +0200 Committer: Fokko Driesprong Committed: Thu Jul 26 10:50:39 2018 +0200 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c06e5fa1/README.md -- diff --git a/README.md b/README.md index 0302495..f5a5133 100644 --- a/README.md +++ b/README.md @@ -157,6 +157,7 @@ Currently **officially** using Airflow: 1. [HBO](http://www.hbo.com/)[[@yiwang](https://github.com/yiwang)] 1. [Healthjump](http://www.healthjump.com/) [[@miscbits](https://github.com/miscbits)] 1. [HelloFresh](https://www.hellofresh.com) [[@tammymendt](https://github.com/tammymendt) & [@davidsbatista](https://github.com/davidsbatista) & [@iuriinedostup](https://github.com/iuriinedostup)] +1. [Hipages](https://www.hipages.com.au/) [[@arihantsurana](https://github.com/arihantsurana)] 1. [Holimetrix](http://holimetrix.com/) [[@thibault-ketterer](https://github.com/thibault-ketterer)] 1. [Hootsuite](https://github.com/hootsuite) 1. [Hostnfly](https://www.hostnfly.com/) [[@CyrilLeMat](https://github.com/CyrilLeMat) & [@pierrechopin](https://github.com/pierrechopin) & [@alexisrosuel](https://github.com/alexisrosuel)]
[jira] [Closed] (AIRFLOW-2781) SLA Usage
[
https://issues.apache.org/jira/browse/AIRFLOW-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG closed AIRFLOW-2781.
--
Resolution: Invalid
> SLA Usage
> -
>
> Key: AIRFLOW-2781
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2781
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.9.0
>Reporter: Xiaodong DENG
>Priority: Minor
>
> Hi, the documentation or information about SLA is quite limited.
> I'm trying to up SLA for my DAGs, but it's not working as I expected/assumed.
> One example is
> {code:java}
> //
> https://github.com/DFStoneburner/airflow-sla-examples/blob/master/basic_sla.py
> t1 = BashOperator(
> task_id='timeout',
> # Set our task up with a 10 second SLA
> sla=timedelta(seconds=10),
> # Sleep 15 seconds to guarantee we miss the SLA
> bash_command='sleep 15',
> dag=dag
> )
> {code}
> Actually I don't see any record in "*Browse -> SLA Misses*" even if this DAG
> can run on schedule.
> For my specific scenario, let's say my schedule interval of the DAG is `_0 3
> * * *`_ (3AM in the morning),
> * if I expect my task is succeeded by 3:30AM, I set
> `*sla=timedelta(minutes=30)*`.
> * if I expect my task is succeeded by 3AM or earlier, I set
> `*sla=timedelta(seconds=0)*`.
> But seems it's not working as I assumed.
>
> May any one kindly advise on this? Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
