[jira] [Created] (AIRFLOW-326) Unclean scheduler restarts result in zombie Task Instances

[George Leslie-Waksman (JIRA)]

George Leslie-Waksman created AIRFLOW-326:
-

 Summary: Unclean scheduler restarts result in zombie Task Instances
 Key: AIRFLOW-326
 URL: https://issues.apache.org/jira/browse/AIRFLOW-326
 Project: Apache Airflow
  Issue Type: Bug
Reporter: George Leslie-Waksman


When a task is running and the scheduler and task both die before the task 
completes execution, the task is not identified as failed. The task appears to 
be in a "Running" state even though it has long since stopped running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-62) XCom push not working reliably

[Martijn Vermaat (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371759#comment-15371759
 ] 

Martijn Vermaat commented on AIRFLOW-62:


I am experiencing the same issue. First task is returning a tuple of strings 
(I'm not pushing explicitely), second task uses it with xcom_pull. It works 
most of the time (and always in my dev setup), but every now and then the 
xcom_pull result is missing. Even though from the logs I can see the first task 
ran succesfully and returned the two strings.

Using Celery with RabbitMQ broker and PostgreSQL backend.

airflow==1.7.1.3
psycopg2==2.6.1
celery==3.1.23


> XCom push not working reliably
> --
>
> Key: AIRFLOW-62
> URL: https://issues.apache.org/jira/browse/AIRFLOW-62
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
>Affects Versions: Airflow 1.7.0
> Environment: Postgres backed Airflow running with Celery inside of 
> the puckel Docker setup.
>Reporter: Alex Papanicolaou
>Assignee: Jeremiah Lowin
> Fix For: Airflow 1.7.0
>
>
> I have a DAG that polls for activity in various data streams from a database 
> and then uploads the activity statuses to a table.  Each of the polling tasks 
> are python operators that once they get the polling result, return a dict as 
> an XCom push.  The dict contains two entries which are strings, one which is 
> a bool, and one which is a datetime object.  There is a final task that pulls 
> all the results and uploads the collective statuses to a table.  I chose this 
> pattern since I figured it might be better to do one collective write 
> operation on all the results.
> Before I moved ahead to the github master branch I was using 1.7.0 from PyPI 
> and this worked fine.  Now that I am on the github master branch, I find that 
> the XCom pushing is unreliable.  The returned values in the logs show up 
> correctly but when doing the XCom pull, I get None for some of the returned 
> values.  Investigating the XCom result in the Webserver also shows nothing 
> there.  But if I rerun a task where the XCom failed, the push works and the 
> XCom result is as it should be.
> Nothing appears to have changed in the codebase so I am at a loss.  Perhaps 
> it really wasn't working before?  How would the backing postgres handle these 
> simultaneous writes?  I can't imagine that would be a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-288) Make system timezone configurable

[George Leslie-Waksman (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371745#comment-15371745
 ] 

George Leslie-Waksman commented on AIRFLOW-288:
---

An alternate solution would be to use timezone aware datetime objects 
throughout airflow.

> Make system timezone configurable
> -
>
> Key: AIRFLOW-288
> URL: https://issues.apache.org/jira/browse/AIRFLOW-288
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Vineet Goel
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-325) [Airflow] dependency on previous task

[Vinod Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371703#comment-15371703
 ] 

Vinod Reddy commented on AIRFLOW-325:
-

Even I am looking for the same kind of parameter.

Thanks
Vinod.

> [Airflow] dependency on previous task
> -
>
> Key: AIRFLOW-325
> URL: https://issues.apache.org/jira/browse/AIRFLOW-325
> Project: Apache Airflow
>  Issue Type: Task
>  Components: executor
>Reporter: Vinod Reddy
>Assignee: Maxime Beauchemin
>Priority: Critical
>
> Hi Maxime,
> I am using depends_on_past = True in a DAG.
> But I need a capability to run the task not only when previous task is 
> succeeded but also failed. Could you please help me achieve this?
> Note: I need dependency on previous task, wanted to start current task only 
> when succeeded or failed. But now depends_on_past = True helps me depend only 
> on previous task's success state.
> I appreciate any help here.
> Thanks
> Vinod.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-325) [Airflow] dependency on previous task

[Vinod Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Reddy updated AIRFLOW-325:

Assignee: Maxime Beauchemin  (was: Vinod Reddy)

> [Airflow] dependency on previous task
> -
>
> Key: AIRFLOW-325
> URL: https://issues.apache.org/jira/browse/AIRFLOW-325
> Project: Apache Airflow
>  Issue Type: Task
>  Components: executor
>Reporter: Vinod Reddy
>Assignee: Maxime Beauchemin
>Priority: Critical
>
> Hi Maxime,
> I am using depends_on_past = True in a DAG.
> But I need a capability to run the task not only when previous task is 
> succeeded but also failed. Could you please help me achieve this?
> Note: I need dependency on previous task, wanted to start current task only 
> when succeeded or failed. But now depends_on_past = True helps me depend only 
> on previous task's success state.
> I appreciate any help here.
> Thanks
> Vinod.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-325) [Airflow] dependency on previous task

[Vinod Reddy (JIRA)]

Vinod Reddy created AIRFLOW-325:
---

 Summary: [Airflow] dependency on previous task
 Key: AIRFLOW-325
 URL: https://issues.apache.org/jira/browse/AIRFLOW-325
 Project: Apache Airflow
  Issue Type: Task
  Components: executor
Reporter: Vinod Reddy
Assignee: Vinod Reddy
Priority: Critical


Hi Maxime,

I am using depends_on_past = True in a DAG.
But I need a capability to run the task not only when previous task is 
succeeded but also failed. Could you please help me achieve this?
Note: I need dependency on previous task, wanted to start current task only 
when succeeded or failed. But now depends_on_past = True helps me depend only 
on previous task's success state.
I appreciate any help here.

Thanks
Vinod.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-323) Should be able to prevent tasks from overlapping across multiple DAG Runs

[Isaac Steele (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371376#comment-15371376
 ] 

Isaac Steele commented on AIRFLOW-323:
--

Hey [~artwr]. Thanks for the reply. My use case is for replicating data, for 
the most part, the replication tasks are very short, however if there is a 
schema change, it takes longer, but we want all the data to continue rolling 
in, regardless of whether 1 table has to take longer because of extra data or 
schema changes.  The depends_on_past doesn't work as-is because it queues up 
all of the runs. If one task takes an hour, but the schedule is set to 15 
minutes, then there will be 4 tasks that will have to run before it catches 
itself up, and then more, should those 4 take more than 15 minutes total. What 
we want is for a task to just be marked as State.SKIPPED, and just let the 
longer-running task complete.

I've added a parameter to do this in my fork, and it works very well, but need 
to write unit tests around it before I can submit a PR. 

We want multiple DAGs to be running at the same time, that's not the issue, 
just individual tasks to not overlap should they happen to not finish in time. 
(Also, we don't want an entire DAG Run to be held up, if 1 task is delayed.)

The "resources" comment in my original post wasn't really a thing we were 
running into was just thinking it could be a thing for someone else, it was 
just the overlapping and queuing of tasks that we were having problems with in 
our project.

(Note that I've also parameterized allowing failed states to work with 
depends_on_past for our own use case as well [AIRFLOW-324], but again need to 
complete my tests before submitting the PR.)

> Should be able to prevent tasks from overlapping across multiple DAG Runs
> -
>
> Key: AIRFLOW-323
> URL: https://issues.apache.org/jira/browse/AIRFLOW-323
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.2
> Environment: 1.7.1.2
>Reporter: Isaac Steele
>Assignee: Isaac Steele
>
> As a the Airflow administrator,
> If a task from a previous DAG Run is still running when the next scheduled 
> run triggers the same task, there should be a way prevent the tasks from 
> overlapping.
> Otherwise the same code could end up running multiple times simultaneously.
> To reproduce:
> 1) Create a DAG with a short scheduled interval
> 2) Create a task in that DAG to run longer than the interval
> Result: Both tasks end up running that the same time.
> This can cause tasks to compete for resources as well as duplicating or 
> overwriting what the other task is doing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-323) Should be able to prevent tasks from overlapping across multiple DAG Runs

[Arthur Wiedmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371291#comment-15371291
 ] 

Arthur Wiedmer commented on AIRFLOW-323:


Hi Isaac, it sounds like there are a couple of things that could help you :
1) You can set max_active_runs for the DAG to 1 to ensure that only one dag run 
is active at a time. In this case, only one dag run will be executed at a time.
2) You can set depend_on_past to True such that this task will not execute 
unless the previous one completes.
3) Finally, you can make this DAG use a pool with one slot, such that this task 
basically takes a lock on this particular resource.

Though ideally, if several tasks are competing for the same resource, you might 
not want to schedule them at a cadence that will introduce contention...

> Should be able to prevent tasks from overlapping across multiple DAG Runs
> -
>
> Key: AIRFLOW-323
> URL: https://issues.apache.org/jira/browse/AIRFLOW-323
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.2
> Environment: 1.7.1.2
>Reporter: Isaac Steele
>Assignee: Isaac Steele
>
> As a the Airflow administrator,
> If a task from a previous DAG Run is still running when the next scheduled 
> run triggers the same task, there should be a way prevent the tasks from 
> overlapping.
> Otherwise the same code could end up running multiple times simultaneously.
> To reproduce:
> 1) Create a DAG with a short scheduled interval
> 2) Create a task in that DAG to run longer than the interval
> Result: Both tasks end up running that the same time.
> This can cause tasks to compete for resources as well as duplicating or 
> overwriting what the other task is doing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-324) Create parameter to allow for any completed state to work with `depends_on_past`.

[Isaac Steele (JIRA)]

Isaac Steele created AIRFLOW-324:


 Summary: Create parameter to allow for any completed state to work 
with `depends_on_past`.
 Key: AIRFLOW-324
 URL: https://issues.apache.org/jira/browse/AIRFLOW-324
 Project: Apache Airflow
  Issue Type: Improvement
Affects Versions: Airflow 1.7.1.2
 Environment: 1.7.1.2
Reporter: Isaac Steele
Assignee: Isaac Steele


As an Airflow admin,
Some tasks should run regardless of whether their previous state was 
successful, as long as it has completed (i.e. SUCCESS, SKIPPED, FAILED, 
UPSTREAM_FAILED). 
If there the depends_on_past parameter is set to True, currently only 
successful and skipped tasks will run. A parameter should exist to allow for 
other completed states.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-323) Should be able to prevent tasks from overlapping across multiple DAG Runs

[Isaac Steele (JIRA)]

Isaac Steele created AIRFLOW-323:


 Summary: Should be able to prevent tasks from overlapping across 
multiple DAG Runs
 Key: AIRFLOW-323
 URL: https://issues.apache.org/jira/browse/AIRFLOW-323
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: Airflow 1.7.1.2
 Environment: 1.7.1.2
Reporter: Isaac Steele
Assignee: Isaac Steele


As a the Airflow administrator,
If a task from a previous DAG Run is still running when the next scheduled run 
triggers the same task, there should be a way prevent the tasks from 
overlapping.
Otherwise the same code could end up running multiple times simultaneously.

To reproduce:
1) Create a DAG with a short scheduled interval
2) Create a task in that DAG to run longer than the interval
Result: Both tasks end up running that the same time.

This can cause tasks to compete for resources as well as duplicating or 
overwriting what the other task is doing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-248) Add Apache license header to all files

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370811#comment-15370811
 ] 

ASF subversion and git services commented on AIRFLOW-248:
-

Commit 15f710fb34cc1164a209464b4c135939c146342d in incubator-airflow's branch 
refs/heads/branch-1.7.2-apache from [~ajayydv]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=15f710f ]

[AIRFLOW-248] Add Apache license header to all files

- Added Apache license header for files with extension (.service, .in, .mako, 
.properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, 
.html, .xml.
- Added/Replaced shebang on all .sh files with portable version - 
#!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.

Closes #1598 from ajayyadava/248


> Add Apache license header to all files
> --
>
> Key: AIRFLOW-248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-248
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Ajay Yadava
>Assignee: Ajay Yadava
> Fix For: Airflow 1.8
>
>
> As part of getting ready for an Apache Release we should add Apache License 
> header to all the files( [mailing list discussion | 
> http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201606.mbox/%3C43C19AB6-9A7F-4351-862E-540D75490396%40gmail.com%3E]
>  ). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-150) setup.py classifiers should be a list not a dict

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370807#comment-15370807
 ] 

ASF subversion and git services commented on AIRFLOW-150:
-

Commit 191c21ee3a3bc2bc68395d39cfc979abb54e3335 in incubator-airflow's branch 
refs/heads/branch-1.7.2-apache from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=191c21e ]

[AIRFLOW-150] setup.py classifiers dict should be list


> setup.py classifiers should be a list not a dict
> 
>
> Key: AIRFLOW-150
> URL: https://issues.apache.org/jira/browse/AIRFLOW-150
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>  Labels: newbie
> Fix For: Airflow 1.8
>
>
> I ran into this issue when trying to release to pypi, setuptools expects 
> classifiers in setup.py to be a list, not a dict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-238) New flask-admin==1.4.1 breaks the Pypi release 1.7.1.2

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370809#comment-15370809
 ] 

ASF subversion and git services commented on AIRFLOW-238:
-

Commit bae76e81b846da2d54cf1d895c57fa3913ca3463 in incubator-airflow's branch 
refs/heads/branch-1.7.2-apache from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=bae76e8 ]

[AIRFLOW-238] Make compatible with flask-admin 1.4.1

The new flask-admin==1.4.1 release on 2016-06-13 breaks the Airflow
release currently in Pypi (1.7.1.2). This fixes the edge case triggered
by this new release.

* Closes #1588 on github


> New flask-admin==1.4.1 breaks the Pypi release 1.7.1.2
> --
>
> Key: AIRFLOW-238
> URL: https://issues.apache.org/jira/browse/AIRFLOW-238
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-9) Create Airflow website

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370808#comment-15370808
 ] 

ASF subversion and git services commented on AIRFLOW-9:
---

Commit 55ac7b58921aaec35f044b3892a4a0a8ef3396a6 in incubator-airflow's branch 
refs/heads/branch-1.7.2-apache from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=55ac7b5 ]

[AIRFLOW-9] Improving docs to meet Apache's standards


> Create Airflow website
> --
>
> Key: AIRFLOW-9
> URL: https://issues.apache.org/jira/browse/AIRFLOW-9
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: project-management
>Reporter: Chris Riccomini
>
> We should set up an Airflow website for:
> http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[01/16] incubator-airflow git commit: [AIRFLOW-150] setup.py classifiers dict should be list

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/branch-1.7.2-apache [created] 27d2d4c79


[AIRFLOW-150] setup.py classifiers dict should be list


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/191c21ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/191c21ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/191c21ee

Branch: refs/heads/branch-1.7.2-apache
Commit: 191c21ee3a3bc2bc68395d39cfc979abb54e3335
Parents: 6f7ea90
Author: Dan Davydov 
Authored: Thu May 19 17:36:56 2016 -0700
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:16:37 2016 +0200

--
 setup.py | 184 ++
 1 file changed, 95 insertions(+), 89 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/191c21ee/setup.py
--
diff --git a/setup.py b/setup.py
index 259828e..2723ed4 100644
--- a/setup.py
+++ b/setup.py
@@ -103,92 +103,98 @@ devel_minreq = devel + mysql + doc + password + s3
 devel_hadoop = devel_minreq + hive + hdfs + webhdfs + kerberos
 devel_all = devel + all_dbs + doc + samba + s3 + slack + crypto + oracle + 
docker
 
-setup(
-name='airflow',
-description='Programmatically author, schedule and monitor data pipelines',
-license='Apache License 2.0',
-version=version,
-packages=find_packages(),
-package_data={'': ['airflow/alembic.ini']},
-include_package_data=True,
-zip_safe=False,
-scripts=['airflow/bin/airflow'],
-install_requires=[
-'alembic>=0.8.3, <0.9',
-'babel>=1.3, <2.0',
-'chartkick>=0.4.2, < 0.5',
-'croniter>=0.3.8, <0.4',
-'dill>=0.2.2, <0.3',
-'python-daemon>=2.1.1, <2.2',
-'flask>=0.10.1, <0.11',
-'flask-admin>=1.4.0, <2.0.0',
-'flask-cache>=0.13.1, <0.14',
-'flask-login==0.2.11',
-'future>=0.15.0, <0.16',
-'funcsigs>=0.4, <1',
-'gunicorn>=19.3.0, <19.4.0',  # 19.4.? seemed to have issues
-'jinja2>=2.7.3, <3.0',
-'markdown>=2.5.2, <3.0',
-'pandas>=0.15.2, <1.0.0',
-'pygments>=2.0.1, <3.0',
-'python-dateutil>=2.3, <3',
-'requests>=2.5.1, <3',
-'setproctitle>=1.1.8, <2',
-'sqlalchemy>=0.9.8',
-'thrift>=0.9.2, <0.10',
-'Flask-WTF==0.12'
-],
-extras_require={
-'all': devel_all,
-'all_dbs': all_dbs,
-'async': async,
-'celery': celery,
-'crypto': crypto,
-'devel': devel_minreq,
-'devel_hadoop': devel_hadoop,
-'doc': doc,
-'docker': docker,
-'druid': druid,
-'gcloud': gcloud,
-'gcp_api': gcp_api,
-'hdfs': hdfs,
-'hive': hive,
-'jdbc': jdbc,
-'mssql': mssql,
-'mysql': mysql,
-'oracle': oracle,
-'postgres': postgres,
-'rabbitmq': rabbitmq,
-'s3': s3,
-'samba': samba,
-'slack': slack,
-'statsd': statsd,
-'vertica': vertica,
-'ldap': ldap,
-'webhdfs': webhdfs,
-'kerberos': kerberos,
-'password': password,
-'github_enterprise': github_enterprise,
-'qds': qds,
-'cloudant': cloudant
-},
-classifiers={
-'Development Status :: 5 - Production/Stable',
-'Environment :: Console',
-'Environment :: Web Environment',
-'Intended Audience :: Developers',
-'Intended Audience :: System Administrators',
-'License :: OSI Approved :: Apache Software License',
-'Programming Language :: Python :: 2.7',
-'Programming Language :: Python :: 3.4',
-'Topic :: System :: Monitoring',
-},
-author='Maxime Beauchemin',
-author_email='maximebeauche...@gmail.com',
-url='https://github.com/airbnb/airflow',
-download_url=(
-'https://github.com/airbnb/airflow/tarball/' + version),
-cmdclass={'test': Tox,
-  'extra_clean': CleanCommand,
-  },
-)
+def do_setup():
+write_version()
+setup(
+name='airflow',
+description='Programmatically author, schedule and monitor data 
pipelines',
+license='Apache License 2.0',
+version=version,
+packages=find_packages(),
+package_data={'': ['airflow/alembic.ini', "airflow/git_version"]},
+include_package_data=True,
+zip_safe=False,
+scripts=['airflow/bin/airflow'],
+install_requires=[
+'alembic>=0.8.3, <0.9',
+'babel>=1.3, <2.0',
+'chartkick>=0.4.2, < 0.5',
+'croniter>=0.3.8, <0.4',
+'dill>=0.2.2, <0.3',
+ 

[04/16] incubator-airflow git commit: [AIRFLOW-238] Make compatible with flask-admin 1.4.1

[bolke]

[AIRFLOW-238] Make compatible with flask-admin 1.4.1

The new flask-admin==1.4.1 release on 2016-06-13 breaks the Airflow
release currently in Pypi (1.7.1.2). This fixes the edge case triggered
by this new release.

* Closes #1588 on github


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/bae76e81
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/bae76e81
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/bae76e81

Branch: refs/heads/branch-1.7.2-apache
Commit: bae76e81b846da2d54cf1d895c57fa3913ca3463
Parents: 55ac7b5
Author: Maxime Beauchemin 
Authored: Tue Jun 14 12:22:05 2016 +0200
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:29:16 2016 +0200

--
 airflow/__init__.py  |  2 +-
 airflow/www/views.py | 16 ++--
 setup.py |  4 ++--
 3 files changed, 9 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/bae76e81/airflow/__init__.py
--
diff --git a/airflow/__init__.py b/airflow/__init__.py
index 900c744..9185c9e 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -20,7 +20,7 @@ in their PYTHONPATH. airflow_login should be based off the
 `airflow.www.login`
 """
 from builtins import object
-__version__ = "1.7.1"
+__version__ = "1.7.1.3"
 
 import logging
 import os

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/bae76e81/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 6e3ff42..c55800a 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -90,14 +90,9 @@ def dag_link(v, c, m, p):
 '{m.dag_id}'.format(**locals()))
 
 
-def log_link(v, c, m, p):
-url = url_for(
-'airflow.log',
-dag_id=m.dag_id,
-task_id=m.task_id,
-execution_date=m.execution_date.isoformat())
+def log_url_formatter(v, c, m, p):
 return Markup(
-''
+''
 ''
 '').format(**locals())
 
@@ -815,7 +810,7 @@ class Airflow(BaseView):
 host = ti.hostname
 log_loaded = False
 
-if socket.gethostname() == host:
+if os.path.exists(loc):
 try:
 f = open(loc)
 log += "".join(f.readlines())
@@ -2100,7 +2095,8 @@ class TaskInstanceModelView(ModelViewOnly):
 'queue', 'pool', 'operator', 'start_date', 'end_date')
 named_filter_urls = True
 column_formatters = dict(
-log=log_link, task_id=task_instance_link,
+log_url=log_url_formatter,
+task_id=task_instance_link,
 hostname=nobr_f,
 state=state_f,
 execution_date=datetime_f,
@@ -2121,7 +2117,7 @@ class TaskInstanceModelView(ModelViewOnly):
 'state', 'dag_id', 'task_id', 'execution_date', 'operator',
 'start_date', 'end_date', 'duration', 'job_id', 'hostname',
 'unixname', 'priority_weight', 'queue', 'queued_dttm', 'try_number',
-'pool', 'log')
+'pool', 'log_url')
 can_delete = True
 page_size = 500
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/bae76e81/setup.py
--
diff --git a/setup.py b/setup.py
index 905af11..864d888 100644
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@ import os
 import sys
 
 # Kept manually in sync with airflow.__version__
-version = '1.7.1'
+version = '1.7.1.3'
 
 
 class Tox(TestCommand):
@@ -123,7 +123,7 @@ def do_setup():
 'dill>=0.2.2, <0.3',
 'python-daemon>=2.1.1, <2.2',
 'flask>=0.10.1, <0.11',
-'flask-admin>=1.4.0, <2.0.0',
+'flask-admin==1.4.1',
 'flask-cache>=0.13.1, <0.14',
 'flask-login==0.2.11',
 'future>=0.15.0, <0.16',

[08/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/static/highcharts.js
--
diff --git a/airflow/www/static/highcharts.js b/airflow/www/static/highcharts.js
deleted file mode 100644
index 1133176..000
--- a/airflow/www/static/highcharts.js
+++ /dev/null
@@ -1,308 +0,0 @@
-/*
- Highcharts JS v4.0.4 (2014-09-02)
-
- (c) 2009-2014 Torstein Honsi
-
- License: www.highcharts.com/license
-*/
-(function(){function r(a,b){var c;a||(a={});for(c in b)a[c]=b[c];return 
a}function w(){var a,b=arguments,c,d={},e=function(a,b){var c,d;typeof 
a!=="object"&&(a={});for(d in b)b.hasOwnProperty(d)&&(c=b[d],a[d]=c& 
c==="object"&(c)!=="[object 
Array]"&!=="renderTo"& 
c.nodeType!=="number"?e(a[d]||{},c):b[d]);return 
a};b[0]===!0&&(d=b[1],b=Array.prototype.slice.call(b,2));c=b.length;for(a=0;a3?k.length%3:0;return
 
e!==Ba?e(a,b,c,d):j+(l?k.substr(0,l)+f:"")+k.substr(l).replace(/(\d{3})(?=\d)/g,"$1"+f)+(h?i+Q(g-k).toFixed(h).slice(2):"")}function
 Ia(a,b){return Array((b||2)+1-String(a).length).join(0)+a}function 
Na(a,b,c){var d=a[b];a[b]=function(){var 
a=Array.prototype.slice.call(arguments);
-a.unshift(d);return c.apply(this,a)}}function Ja(a,b){for(var 
c="{",d=!1,e,f,g,h,i,j=[];(c=a.indexOf(c))!==-1;){e=a.slice(0,c);if(d){f=e.split(":");g=f.shift().split(".");i=g.length;e=b;for(h=0;h-1?h.thousandsSep:""))):e=cb(f,e)}j.push(e);a=a.slice(c+1);c=(d=!d)?"}":"{"}j.push(a);return
 j.join("")}function mb(a){return V.pow(10,U(V.log(a)/
-V.LN10))}function nb(a,b,c,d){var 
e,c=p(c,1);e=a/c;b||(b=[1,2,2.5,5,10],d===!1&&(c===1?b=[1,2,5,10]:c<=0.1&&(b=[1/c])));for(d=0;dc&&(c=a[b]);return c}
-function Pa(a,b){for(var c in 
a)a[c]&[c]!==b&[c].destroy&[c].destroy(),delete a[c]}function 
Qa(a){db||(db=$(Ka));a&(a);db.innerHTML=""}function ea(a){return 
parseFloat(a.toPrecision(14))}function Ra(a,b){va=p(a,b.animation)}function 
Bb(){var 
a=E.global.useUTC,b=a?"getUTC":"get",c=a?"setUTC":"set";Da=E.global.Date||window.Date;Sa=(a&||0)*6E4;eb=a?Da.UTC:function(a,b,c,g,h,i){return(new
 Da(a,b,p(c,1),p(g,0),p(h,0),p(i,0))).getTime()};pb=b+"Minutes";qb=b+
-"Hours";rb=b+"Day";Xa=b+"Date";fb=b+"Month";gb=b+"FullYear";Cb=c+"Minutes";Db=c+"Hours";sb=c+"Date";Eb=c+"Month";Fb=c+"FullYear"}function
 S(){}function 
Ta(a,b,c,d){this.axis=a;this.pos=b;this.type=c||"";this.isNew=!0;!c&&!d&()}function
 na(){this.init.apply(this,arguments)}function 
Ya(){this.init.apply(this,arguments)}function Gb(a,b,c,d,e){var 
f=a.chart.inverted;this.axis=a;this.isNegative=c;this.options=b;this.x=d;this.total=null;this.points={};this.stack=e;this.alignOptions={align:b.align||
-(f?c?"left":"right":"center"),verticalAlign:b.verticalAlign||(f?"middle":c?"bottom":"top"),y:p(b.y,f?4:c?14:-6),x:p(b.x,f?c?-6:6:0)};this.textAlign=b.textAlign||(f?c?"right":"left":"center")}var
 
u,x=document,G=window,V=Math,v=V.round,U=V.floor,La=V.ceil,t=V.max,L=V.min,Q=V.abs,aa=V.cos,fa=V.sin,oa=V.PI,Ea=oa*2/360,wa=navigator.userAgent,Hb=G.opera,Aa=/msie/i.test(wa)&&!Hb,hb=x.documentMode===8,tb=/AppleWebKit/.test(wa),Ua=/Firefox/.test(wa),Ib=/(Mobile|Android|Windows
 Phone)/.test(wa),xa="http://www.w3.org/2000/svg;,

[03/16] incubator-airflow git commit: [AIRFLOW-9] Improving docs to meet Apache's standards

[bolke]

[AIRFLOW-9] Improving docs to meet Apache's standards


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/55ac7b58
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/55ac7b58
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/55ac7b58

Branch: refs/heads/branch-1.7.2-apache
Commit: 55ac7b58921aaec35f044b3892a4a0a8ef3396a6
Parents: 4affece
Author: Maxime Beauchemin 
Authored: Tue May 10 15:49:22 2016 -0700
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:25:04 2016 +0200

--
 docs/conf.py   |   4 +-
 docs/img/apache.jpg| Bin 0 -> 43364 bytes
 docs/img/incubator.jpg | Bin 0 -> 91227 bytes
 docs/index.rst |  13 ++-
 docs/license.rst   | 211 
 docs/project.rst   |  59 +
 6 files changed, 282 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/55ac7b58/docs/conf.py
--
diff --git a/docs/conf.py b/docs/conf.py
index 3bb2d9f..b0b5195 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -54,7 +54,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Airflow'
-copyright = u'2014, Maxime Beauchemin, Airbnb'
+#copyright = u''
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -177,7 +177,7 @@ html_use_index = True
 #html_show_sphinx = True
 
 # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
+html_show_copyright = False
 
 # If true, an OpenSearch description file will be output, and all pages will
 # contain a  tag referring to it.  The value of this option must be the

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/55ac7b58/docs/img/apache.jpg
--
diff --git a/docs/img/apache.jpg b/docs/img/apache.jpg
new file mode 100644
index 000..312251f
Binary files /dev/null and b/docs/img/apache.jpg differ

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/55ac7b58/docs/img/incubator.jpg
--
diff --git a/docs/img/incubator.jpg b/docs/img/incubator.jpg
new file mode 100644
index 000..6f34a85
Binary files /dev/null and b/docs/img/incubator.jpg differ

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/55ac7b58/docs/index.rst
--
diff --git a/docs/index.rst b/docs/index.rst
index c4ccca5..70f9355 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,7 +1,10 @@
+
 .. image:: img/pin_large.png
-:width: 70
+:width: 100
+.. image:: img/incubator.jpg
+:width: 150
 
-Airflow Documentation
+Apache Airflow (incubating) Documentation
 
 
 Airflow is a platform to programmatically author, schedule and monitor
@@ -19,6 +22,7 @@ versionable, testable, and collaborative.
 
 
 
+
 .. image:: img/airflow.gif
 
 
@@ -48,13 +52,16 @@ than a database structure would be. Airflow workflows are 
expected to look
 similar from a run to the next, this allows for clarity around
 unit of work and continuity.
 
+
 Content
 ---
 .. toctree::
 :maxdepth: 4
 
-installation
+project
+license
 start
+installation
 tutorial
 configuration
 ui

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/55ac7b58/docs/license.rst
--
diff --git a/docs/license.rst b/docs/license.rst
new file mode 100644
index 000..9da26c0
--- /dev/null
+++ b/docs/license.rst
@@ -0,0 +1,211 @@
+License
+===
+
+.. image:: img/apache.jpg
+:width: 150
+
+::
+
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+
+   "Licensor" shall mean the copyright owner or entity authorized by
+   the copyright owner that is granting the License.
+
+   "Legal Entity" shall mean the union of the acting entity and all
+   other entities that control, are controlled by, or are under common
+   control with that entity. For the purposes of this definition,
+   "control" means (i) the power, direct or indirect, to cause the
+   direction or management of such entity, whether by 

[06/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/static/nv.d3.js
--
diff --git a/airflow/www/static/nv.d3.js b/airflow/www/static/nv.d3.js
new file mode 100644
index 000..f00492a
--- /dev/null
+++ b/airflow/www/static/nv.d3.js
@@ -0,0 +1,14241 @@
+(function(){
+
+var nv = window.nv || {};
+
+
+nv.version = '1.1.11b';
+nv.dev = true //set false when in production
+
+window.nv = nv;
+
+nv.tooltip = {}; // For the tooltip system
+nv.utils = {}; // Utility subsystem
+nv.models = {}; //stores all the possible models/components
+nv.charts = {}; //stores all the ready to use charts
+nv.graphs = []; //stores all the graphs currently on the page
+nv.logs = {}; //stores some statistics and potential error messages
+
+nv.dispatch = d3.dispatch('render_start', 'render_end');
+
+// *
+//  Development render timers - disabled if dev = false
+
+if (nv.dev) {
+  nv.dispatch.on('render_start', function(e) {
+nv.logs.startTime = +new Date();
+  });
+
+  nv.dispatch.on('render_end', function(e) {
+nv.logs.endTime = +new Date();
+nv.logs.totalTime = nv.logs.endTime - nv.logs.startTime;
+nv.log('total', nv.logs.totalTime); // used for development, to keep track 
of graph generation times
+  });
+}
+
+// 
+//  Public Core NV functions
+
+// Logs all arguments, and returns the last so you can test things in place
+// Note: in IE8 console.log is an object not a function, and if modernizr is 
used
+// then calling Function.prototype.bind with with anything other than a 
function
+// causes a TypeError to be thrown.
+nv.log = function() {
+  if (nv.dev && console.log && console.log.apply)
+console.log.apply(console, arguments)
+  else if (nv.dev && typeof console.log == "function" && 
Function.prototype.bind) {
+var log = Function.prototype.bind.call(console.log, console);
+log.apply(console, arguments);
+  }
+  return arguments[arguments.length - 1];
+};
+
+
+nv.render = function render(step) {
+  step = step || 1; // number of graphs to generate in each timeout loop
+
+  nv.render.active = true;
+  nv.dispatch.render_start();
+
+  setTimeout(function() {
+var chart, graph;
+
+for (var i = 0; i < step && (graph = nv.render.queue[i]); i++) {
+  chart = graph.generate();
+  if (typeof graph.callback == typeof(Function)) graph.callback(chart);
+  nv.graphs.push(chart);
+}
+
+nv.render.queue.splice(0, i);
+
+if (nv.render.queue.length) setTimeout(arguments.callee, 0);
+else { nv.render.active = false; nv.dispatch.render_end(); }
+  }, 0);
+};
+
+nv.render.active = false;
+nv.render.queue = [];
+
+nv.addGraph = function(obj) {
+  if (typeof arguments[0] === typeof(Function))
+obj = {generate: arguments[0], callback: arguments[1]};
+
+  nv.render.queue.push(obj);
+
+  if (!nv.render.active) nv.render();
+};
+
+nv.identity = function(d) { return d; };
+
+nv.strip = function(s) { return s.replace(/(\s|&)/g,''); };
+
+function daysInMonth(month,year) {
+  return (new Date(year, month+1, 0)).getDate();
+}
+
+function d3_time_range(floor, step, number) {
+  return function(t0, t1, dt) {
+var time = floor(t0), times = [];
+if (time < t0) step(time);
+if (dt > 1) {
+  while (time < t1) {
+var date = new Date(+time);
+if ((number(date) % dt === 0)) times.push(date);
+step(time);
+  }
+} else {
+  while (time < t1) { times.push(new Date(+time)); step(time); }
+}
+return times;
+  };
+}
+
+d3.time.monthEnd = function(date) {
+  return new Date(date.getFullYear(), date.getMonth(), 0);
+};
+
+d3.time.monthEnds = d3_time_range(d3.time.monthEnd, function(date) {
+date.setUTCDate(date.getUTCDate() + 1);
+date.setDate(daysInMonth(date.getMonth() + 1, date.getFullYear()));
+  }, function(date) {
+return date.getMonth();
+  }
+);
+
+/* Utility class to handle creation of an interactive layer.
+This places a rectangle on top of the chart. When you mouse move over it, it 
sends a dispatch
+containing the X-coordinate. It can also render a vertical line where the 
mouse is located.
+
+dispatch.elementMousemove is the important event to latch onto.  It is fired 
whenever the mouse moves over
+the rectangle. The dispatch is given one object which contains the mouseX/Y 
location.
+It also has 'pointXValue', which is the conversion of mouseX to the x-axis 
scale.
+*/
+nv.interactiveGuideline = function() {
+   "use strict";
+   var tooltip = nv.models.tooltip();
+   //Public settings
+   var width = null
+   , height = null
+//Please pass in the bounding chart's top and left margins
+//This is important for calculating the correct mouseX/Y positions.
+   , margin = {left: 0, top: 0}
+   , xScale = d3.scale.linear()
+   , yScale = d3.scale.linear()
+   , dispatch 

[09/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/static/gantt-chart-d3v2.js
--
diff --git a/airflow/www/static/gantt-chart-d3v2.js 
b/airflow/www/static/gantt-chart-d3v2.js
new file mode 100644
index 000..b8924f9
--- /dev/null
+++ b/airflow/www/static/gantt-chart-d3v2.js
@@ -0,0 +1,247 @@
+/**
+ * @author Dimitry Kudrayvtsev
+ * @version 2.1
+ * @modifiedby Maxime Beauchemin
+ */
+
+d3.gantt = function() {
+  var FIT_TIME_DOMAIN_MODE = "fit";
+  var FIXED_TIME_DOMAIN_MODE = "fixed";
+  var tip = d3.tip()
+.attr('class', 'd3-tip')
+.offset([-10, 0])
+.html(function(d) {
+  var s = ""
+  s += "";
+  s += "start:end:duration:"
+  s += " "
+  s += d.isoStart + "";
+  s += d.isoEnd + "";
+  s += d.duration + "";
+  s += "";
+  s += "";
+  return s;
+  })
+
+  var margin = {
+top : 20,
+right : 40,
+bottom : 20,
+left : 150
+  };
+  var selector = 'body';
+  var timeDomainStart = d3.time.day.offset(new Date(),-3);
+  var timeDomainEnd = d3.time.hour.offset(new Date(),+3);
+  var timeDomainMode = FIT_TIME_DOMAIN_MODE;// fixed or fit
+  var taskTypes = [];
+  var taskStatus = [];
+  var height = document.body.clientHeight - margin.top - margin.bottom-5;
+  var width = $('.gantt').width() - margin.right - margin.left-5;
+
+  var tickFormat = "%H:%M";
+
+  var keyFunction = function(d) {
+return d.startDate + d.taskName + d.endDate;
+  };
+
+  var rectTransform = function(d) {
+return "translate(" + x(d.startDate) + "," + y(d.taskName) + ")";
+  };
+
+  var x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([ 
0, width ]).clamp(true);
+
+  var y = d3.scale.ordinal().domain(taskTypes).rangeRoundBands([ 0, height - 
margin.top - margin.bottom ], .1);
+
+  var xAxis = 
d3.svg.axis().scale(x).orient("bottom").tickFormat(d3.time.format(tickFormat)).tickSubdivide(true)
+  .tickSize(8).tickPadding(8);
+
+  var yAxis = d3.svg.axis().scale(y).orient("left").tickSize(0);
+
+  var initTimeDomain = function(tasks) {
+if (timeDomainMode === FIT_TIME_DOMAIN_MODE) {
+  if (tasks === undefined || tasks.length < 1) {
+timeDomainStart = d3.time.day.offset(new Date(), -3);
+timeDomainEnd = d3.time.hour.offset(new Date(), +3);
+return;
+  }
+  tasks.sort(function(a, b) {
+return a.endDate - b.endDate;
+  });
+  timeDomainEnd = tasks[tasks.length - 1].endDate;
+  tasks.sort(function(a, b) {
+return a.startDate - b.startDate;
+  });
+  timeDomainStart = tasks[0].startDate;
+}
+  };
+
+  var initAxis = function() {
+x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([ 0, 
width ]).clamp(true);
+y = d3.scale.ordinal().domain(taskTypes).rangeRoundBands([ 0, height - 
margin.top - margin.bottom ], .1);
+xAxis = 
d3.svg.axis().scale(x).orient("bottom").tickFormat(d3.time.format(tickFormat)).tickSubdivide(true)
+.tickSize(8).tickPadding(8);
+
+yAxis = d3.svg.axis().scale(y).orient("left").tickSize(0);
+  };
+
+  function gantt(tasks) {
+
+initTimeDomain(tasks);
+initAxis();
+
+var svg = d3.select(selector)
+.append("svg")
+.attr("class", "chart")
+.attr("width", width + margin.left + margin.right)
+.attr("height", height + margin.top + margin.bottom)
+.append("g")
+.attr("class", "gantt-chart")
+.attr("width", width + margin.left + margin.right)
+.attr("height", height + margin.top + margin.bottom)
+.attr("transform", "translate(" + margin.left + ", " + margin.top + ")");
+
+svg.selectAll(".chart")
+.data(tasks, keyFunction).enter()
+.append("rect")
+.on('mouseover', tip.show)
+.on('mouseout', tip.hide)
+.on('click', function(d) {
+  call_modal(d.taskName, d.executionDate);
+})
+.attr("class", function(d){
+  if(taskStatus[d.status] == null){ return "bar";}
+  return taskStatus[d.status];
+})
+.attr("y", 0)
+.attr("transform", rectTransform)
+.attr("height", function(d) { return y.rangeBand(); })
+.attr("width", function(d) {
+  return d3.max([x(d.endDate) - x(d.startDate), 1]);
+});
+
+
+svg.append("g")
+.attr("class", "x axis")
+.attr("transform", "translate(0, " + (height - margin.top - margin.bottom) 
+ ")")
+.transition()
+.call(xAxis);
+
+svg.append("g").attr("class", "y axis").transition().call(yAxis);
+svg.call(tip);
+
+return gantt;
+
+  };
+
+  gantt.redraw = function(tasks) {
+
+initTimeDomain(tasks);
+initAxis();
+
+var svg = d3.select(".chart");
+
+var ganttChartGroup = svg.select(".gantt-chart");
+var rect = ganttChartGroup.selectAll("rect").data(tasks, keyFunction);
+
+rect.enter()
+.insert("rect",":first-child")
+.attr("rx", 5)
+.attr("ry", 5)
+.attr("class", function(d){
+  if(taskStatus[d.status] == null){ return "bar";}

[07/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/static/nv.d3.css
--
diff --git a/airflow/www/static/nv.d3.css b/airflow/www/static/nv.d3.css
new file mode 100644
index 000..cae8348
--- /dev/null
+++ b/airflow/www/static/nv.d3.css
@@ -0,0 +1,769 @@
+
+/
+ * HTML CSS
+ */
+
+
+.chartWrap {
+  margin: 0;
+  padding: 0;
+  overflow: hidden;
+}
+
+/
+  Box shadow and border radius styling
+*/
+.nvtooltip.with-3d-shadow, .with-3d-shadow .nvtooltip {
+  -moz-box-shadow: 0 5px 10px rgba(0,0,0,.2);
+  -webkit-box-shadow: 0 5px 10px rgba(0,0,0,.2);
+  box-shadow: 0 5px 10px rgba(0,0,0,.2);
+
+  -webkit-border-radius: 6px;
+  -moz-border-radius: 6px;
+  border-radius: 6px;
+}
+
+/
+ * TOOLTIP CSS
+ */
+
+.nvtooltip {
+  position: absolute;
+  background-color: rgba(255,255,255,1.0);
+  padding: 1px;
+  border: 1px solid rgba(0,0,0,.2);
+  z-index: 1;
+
+  font-family: Arial;
+  font-size: 13px;
+  text-align: left;
+  pointer-events: none;
+
+  white-space: nowrap;
+
+  -webkit-touch-callout: none;
+  -webkit-user-select: none;
+  -khtml-user-select: none;
+  -moz-user-select: none;
+  -ms-user-select: none;
+  user-select: none;
+}
+
+/*Give tooltips that old fade in transition by
+putting a "with-transitions" class on the container div.
+*/
+.nvtooltip.with-transitions, .with-transitions .nvtooltip {
+  transition: opacity 250ms linear;
+  -moz-transition: opacity 250ms linear;
+  -webkit-transition: opacity 250ms linear;
+
+  transition-delay: 250ms;
+  -moz-transition-delay: 250ms;
+  -webkit-transition-delay: 250ms;
+}
+
+.nvtooltip.x-nvtooltip,
+.nvtooltip.y-nvtooltip {
+  padding: 8px;
+}
+
+.nvtooltip h3 {
+  margin: 0;
+  padding: 4px 14px;
+  line-height: 18px;
+  font-weight: normal;
+  background-color: rgba(247,247,247,0.75);
+  text-align: center;
+
+  border-bottom: 1px solid #ebebeb;
+
+  -webkit-border-radius: 5px 5px 0 0;
+  -moz-border-radius: 5px 5px 0 0;
+  border-radius: 5px 5px 0 0;
+}
+
+.nvtooltip p {
+  margin: 0;
+  padding: 5px 14px;
+  text-align: center;
+}
+
+.nvtooltip span {
+  display: inline-block;
+  margin: 2px 0;
+}
+
+.nvtooltip table {
+  margin: 6px;
+  border-spacing:0;
+}
+
+
+.nvtooltip table td {
+  padding: 2px 9px 2px 0;
+  vertical-align: middle;
+}
+
+.nvtooltip table td.key {
+  font-weight:normal;
+}
+.nvtooltip table td.value {
+  text-align: right;
+  font-weight: bold;
+}
+
+.nvtooltip table tr.highlight td {
+  padding: 1px 9px 1px 0;
+  border-bottom-style: solid;
+  border-bottom-width: 1px;
+  border-top-style: solid;
+  border-top-width: 1px;
+}
+
+.nvtooltip table td.legend-color-guide div {
+  width: 8px;
+  height: 8px;
+  vertical-align: middle;
+}
+
+.nvtooltip .footer {
+  padding: 3px;
+  text-align: center;
+}
+
+
+.nvtooltip-pending-removal {
+  position: absolute;
+  pointer-events: none;
+}
+
+
+/
+ * SVG CSS
+ */
+
+
+svg {
+  -webkit-touch-callout: none;
+  -webkit-user-select: none;
+  -khtml-user-select: none;
+  -moz-user-select: none;
+  -ms-user-select: none;
+  user-select: none;
+  /* Trying to get SVG to act like a greedy block in all browsers */
+  display: block;
+  width:100%;
+  height:100%;
+}
+
+
+svg text {
+  font: normal 12px Arial;
+}
+
+svg .title {
+ font: bold 14px Arial;
+}
+
+.nvd3 .nv-background {
+  fill: white;
+  fill-opacity: 0;
+  /*
+  pointer-events: none;
+  */
+}
+
+.nvd3.nv-noData {
+  font-size: 18px;
+  font-weight: bold;
+}
+
+
+/**
+*  Brush
+*/
+
+.nv-brush .extent {
+  fill-opacity: .125;
+  shape-rendering: crispEdges;
+}
+
+
+
+/**
+*  Legend
+*/
+
+.nvd3 .nv-legend .nv-series {
+  cursor: pointer;
+}
+
+.nvd3 .nv-legend .disabled circle {
+  fill-opacity: 0;
+}
+
+
+
+/**
+*  Axes
+*/
+.nvd3 .nv-axis {
+  pointer-events:none;
+}
+
+.nvd3 .nv-axis path {
+  fill: none;
+  stroke: #000;
+  stroke-opacity: .75;
+  shape-rendering: crispEdges;
+}
+
+.nvd3 .nv-axis path.domain {
+  stroke-opacity: .75;
+}
+
+.nvd3 .nv-axis.nv-x path.domain {
+  stroke-opacity: 0;
+}
+
+.nvd3 .nv-axis line {
+  fill: none;
+  stroke: #e5e5e5;
+  shape-rendering: crispEdges;
+}
+
+.nvd3 .nv-axis .zero line,
+/*this selector may not be necessary*/ .nvd3 .nv-axis line.zero {
+  stroke-opacity: .75;
+}
+
+.nvd3 .nv-axis .nv-axisMaxMin text {
+  font-weight: bold;
+}
+
+.nvd3 .x  .nv-axis .nv-axisMaxMin text,
+.nvd3 .x2 .nv-axis .nv-axisMaxMin text,
+.nvd3 .x3 .nv-axis .nv-axisMaxMin text {
+  text-anchor: middle
+}
+
+
+
+/**
+*  Brush
+*/
+
+.nv-brush .resize path {
+  fill: #eee;
+  stroke: #666;
+}
+
+
+
+/**
+*  Bars
+*/
+
+.nvd3 .nv-bars .negative rect {
+zfill: brown;
+}
+
+.nvd3 .nv-bars rect {
+  zfill: steelblue;
+  fill-opacity: .75;
+
+  transition: fill-opacity 250ms linear;
+  -moz-transition: fill-opacity 250ms linear;
+  -webkit-transition: fill-opacity 250ms linear;
+}
+
+.nvd3 

[14/16] incubator-airflow git commit: [AIRFLOW-248] Add Apache license header to all files

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/airflow/www/static/heatmap-canvas.js
--
diff --git a/airflow/www/static/heatmap-canvas.js 
b/airflow/www/static/heatmap-canvas.js
index ccea814..01ee471 100644
--- a/airflow/www/static/heatmap-canvas.js
+++ b/airflow/www/static/heatmap-canvas.js
@@ -1,3 +1,22 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
 
 /**
  * This plugin extends Highcharts in two ways:

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/airflow/www/static/jquery.dataTables.css
--
diff --git a/airflow/www/static/jquery.dataTables.css 
b/airflow/www/static/jquery.dataTables.css
index 4e6fbe3..6a2e88d 100644
--- a/airflow/www/static/jquery.dataTables.css
+++ b/airflow/www/static/jquery.dataTables.css
@@ -1,3 +1,22 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
+
 /*
  * Table styles
  */

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/airflow/www/static/main.css
--
diff --git a/airflow/www/static/main.css b/airflow/www/static/main.css
index aab0683..a394984 100644
--- a/airflow/www/static/main.css
+++ b/airflow/www/static/main.css
@@ -1,16 +1,21 @@
 /**
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied. See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*/
 
 body { padding-top: 70px; }
 a.navbar-brand span {

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/airflow/www/static/mode-sql.js
--
diff --git a/airflow/www/static/mode-sql.js b/airflow/www/static/mode-sql.js
index 66c50c1..6cdfad5 100644
--- a/airflow/www/static/mode-sql.js
+++ b/airflow/www/static/mode-sql.js
@@ -1,3 +1,22 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); 

[15/16] incubator-airflow git commit: [AIRFLOW-248] Add Apache license header to all files

[bolke]

[AIRFLOW-248] Add Apache license header to all files

- Added Apache license header for files with extension (.service, .in, .mako, 
.properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, 
.html, .xml.
- Added/Replaced shebang on all .sh files with portable version - 
#!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.

Closes #1598 from ajayyadava/248


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/15f710fb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/15f710fb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/15f710fb

Branch: refs/heads/branch-1.7.2-apache
Commit: 15f710fb34cc1164a209464b4c135939c146342d
Parents: 6e60664
Author: Ajay Yadav 
Authored: Tue Jun 21 08:15:42 2016 -0700
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:39:58 2016 +0200

--
 .codecov.yml| 14 ++
 .coveragerc | 13 +
 .landscape.yml  | 13 +
 .travis.yml | 13 +
 MANIFEST.in | 17 ++--
 airflow/alembic.ini | 13 +
 airflow/bin/__init__.py | 14 ++
 airflow/bin/airflow_scheduler_autorestart.sh| 13 +
 airflow/bin/cli.py  | 14 ++
 airflow/contrib/__init__.py | 14 ++
 airflow/contrib/auth/__init__.py| 14 ++
 airflow/contrib/auth/backends/__init__.py   | 14 ++
 airflow/contrib/auth/backends/kerberos_auth.py  | 14 ++
 airflow/contrib/auth/backends/ldap_auth.py  | 14 ++
 airflow/contrib/auth/backends/password_auth.py  | 14 ++
 .../example_dags/example_qubole_operator.py | 14 ++
 airflow/contrib/executors/__init__.py   | 14 ++
 airflow/contrib/executors/mesos_executor.py | 14 ++
 .../operators/bigquery_check_operator.py| 14 ++
 airflow/contrib/operators/bigquery_operator.py  | 14 ++
 .../contrib/operators/bigquery_to_bigquery.py   | 14 ++
 airflow/contrib/operators/bigquery_to_gcs.py| 14 ++
 .../contrib/operators/gcs_download_operator.py  | 14 ++
 airflow/contrib/operators/gcs_to_bq.py  | 14 ++
 airflow/contrib/operators/mysql_to_gcs.py   | 14 ++
 airflow/contrib/operators/qubole_operator.py| 14 ++
 .../contrib/operators/ssh_execute_operator.py   | 14 ++
 airflow/contrib/operators/vertica_operator.py   | 14 ++
 airflow/contrib/operators/vertica_to_hive.py| 14 ++
 .../contrib/plugins/metastore_browser/main.py   | 14 ++
 .../templates/metastore_browser/base.html   | 17 
 .../templates/metastore_browser/db.html | 17 
 .../templates/metastore_browser/dbs.html| 17 
 .../templates/metastore_browser/table.html  | 17 
 airflow/example_dags/__init__.py| 14 ++
 airflow/example_dags/entrypoint.sh  | 13 +
 airflow/example_dags/subdags/__init__.py| 14 ++
 airflow/example_dags/subdags/subdag.py  | 14 ++
 airflow/example_dags/tutorial.py| 14 ++
 airflow/executors/__init__.py   | 14 ++
 airflow/executors/local_executor.py | 14 ++
 airflow/executors/sequential_executor.py| 14 ++
 airflow/macros/hive.py  | 14 ++
 airflow/migrations/__init__.py  | 14 ++
 airflow/migrations/env.py   | 14 ++
 airflow/migrations/script.py.mako   | 13 +
 .../versions/13eb55f81627_for_compatibility.py  | 14 ++
 .../1507a7289a2f_create_is_encrypted.py | 14 ++
 ...09e3_add_is_encrypted_column_to_variable_.py | 14 ++
 .../versions/1b38cef5b76e_add_dagrun.py | 14 ++
 .../versions/2e541a1dcfed_task_duration.py  | 14 ++
 .../versions/2e82aab8ef20_rename_user_table.py  | 14 ++
 ...e90f54d61_more_logging_into_task_isntance.py | 14 ++
 .../versions/40e67319e3a9_dagrun_config.py  | 14 ++
 .../versions/4446e08588_dagrun_start_end.py | 14 ++
 .../502898887f84_adding_extra_to_log.py | 14 ++
 .../versions/52d714495f0_job_id_indices.py  | 14 ++
 .../561833c1c74b_add_password_column_to_user.py | 14 ++
 airflow/migrations/versions/__init__.py | 14 ++
 ...896_add_a_column_to_track_the_encryption_.py | 14 ++
 

[12/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

[AIRFLOW-6] Remove dependency on Highcharts

Highcharts' license is not compatible with the Apache 2.0
license. This patch removes Highcharts in favor of d3,
however some charts are not supported anymore.

* This brings Maxime Beauchemin's work to master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6e606646
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6e606646
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6e606646

Branch: refs/heads/branch-1.7.2-apache
Commit: 6e60664672009acdddae20360fb245986ba058ac
Parents: bae76e8
Author: Bolke de Bruin 
Authored: Mon Jun 20 14:19:34 2016 +0200
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:35:14 2016 +0200

--
 `|  2347 
 airflow/bin/airflow  |13 +
 airflow/executors/base_executor.py   |14 +
 airflow/executors/celery_executor.py |14 +
 airflow/hooks/__init__.py|33 +
 airflow/hooks/dbapi_hook.py  |13 +
 airflow/hooks/jdbc_hook.py   | 3 +-
 airflow/hooks/oracle_hook.py | 1 +
 airflow/www/app.py   | 4 +-
 airflow/www/blueprints.py| 6 -
 airflow/www/static/d3.tip.v0.6.3.js  |   280 +
 airflow/www/static/d3.v3.min.js  |10 +-
 airflow/www/static/gantt-chart-d3v2.js   |   247 +
 airflow/www/static/gantt.css |38 +
 airflow/www/static/highcharts-more.js|53 -
 airflow/www/static/highcharts.js |   308 -
 airflow/www/static/nv.d3.css |   769 ++
 airflow/www/static/nv.d3.js  | 14241 
 airflow/www/static/nvd3.tar.gz   |   Bin 0 -> 328377 bytes
 airflow/www/templates/airflow/chart.html |37 +-
 airflow/www/templates/airflow/dag.html   | 4 +-
 airflow/www/templates/airflow/gantt.html |86 +-
 airflow/www/templates/airflow/highchart.html |   183 -
 airflow/www/templates/airflow/nvd3.html  |   175 +
 airflow/www/utils.py |11 +-
 airflow/www/views.py |   342 +-
 setup.py |23 +-
 tests/core.py|19 -
 28 files changed, 18364 insertions(+), 910 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/`
--
diff --git a/` b/`
new file mode 100644
index 000..6331805
--- /dev/null
+++ b/`
@@ -0,0 +1,2347 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import sys
+
+import os
+import socket
+import importlib
+
+from functools import wraps
+from datetime import datetime, timedelta
+import dateutil.parser
+import copy
+from itertools import chain, product
+
+from past.utils import old_div
+from past.builtins import basestring
+
+import inspect
+import traceback
+
+import sqlalchemy as sqla
+from sqlalchemy import or_, desc, and_
+
+
+from flask import redirect, url_for, request, Markup, Response, current_app, 
render_template
+from flask_admin import BaseView, expose, AdminIndexView
+from flask_admin.contrib.sqla import ModelView
+from flask_admin.actions import action
+from flask_login import flash
+from flask._compat import PY2
+
+import jinja2
+import markdown
+import json
+
+from wtforms import (
+Form, SelectField, TextAreaField, PasswordField, StringField)
+
+from pygments import highlight, lexers
+from pygments.formatters import HtmlFormatter
+
+import airflow
+from airflow import configuration as conf
+from airflow import models
+from airflow import settings
+from airflow.exceptions import AirflowException
+from airflow.settings import Session
+from airflow.models import XCom
+
+from airflow.utils.json import json_ser
+from airflow.utils.state import State
+from airflow.utils.db import provide_session
+from airflow.utils.helpers import alchemy_to_dict
+from airflow.utils import logging as log_utils
+from airflow.www import utils as wwwutils
+from airflow.www.forms import DateTimeForm, DateTimeWithNumRunsForm
+
+QUERY_LIMIT = 

[10/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/static/d3.v3.min.js
--
diff --git a/airflow/www/static/d3.v3.min.js b/airflow/www/static/d3.v3.min.js
index 88550ae..77615c0 100644
--- a/airflow/www/static/d3.v3.min.js
+++ b/airflow/www/static/d3.v3.min.js
@@ -1,5 +1,5 @@
-!function(){function n(n,t){return t>n?-1:n>t?1:n>=t?0:0/0}function 
t(n){return null!=n&&!isNaN(n)}function 
e(n){return{left:function(t,e,r,u){for(arguments.length<3&&(r=0),arguments.length<4&&(u=t.length);u>r;){var
 i=r+u>>>1;n(t[i],e)<0?r=i+1:u=i}return 
r},right:function(t,e,r,u){for(arguments.length<3&&(r=0),arguments.length<4&&(u=t.length);u>r;){var
 i=r+u>>>1;n(t[i],e)>0?u=i:r=i+1}return r}}}function r(n){return 
n.length}function u(n){for(var t=1;n*t%1;)t*=10;return t}function 
i(n,t){try{for(var e in 
t)Object.defineProperty(n.prototype,e,{value:t[e],enumerable:!1})}catch(r){n.prototype=t}}function
 o(){}function a(n){return ia+n in this}function c(n){return n=ia+n,n in 
this& this[n]}function s(){var n=[];return 
this.forEach(function(t){n.push(t)}),n}function l(){var n=0;for(var t in 
this)t.charCodeAt(0)===oa&&++n;return n}function f(){for(var n in 
this)if(n.charCodeAt(0)===oa)return!1;return!0}function h(){}function 
g(n,t,e){return function(){var r=e.apply(t,arguments);retu
 rn r===t?n:r}}function p(n,t){if(t in n)return 
t;t=t.charAt(0).toUpperCase()+t.substring(1);for(var 
e=0,r=aa.length;r>e;++e){var u=aa[e]+t;if(u in n)return u}}function 
v(){}function d(){}function m(n){function t(){for(var 
t,r=e,u=-1,i=r.length;++ue;e++)for(var 
u,i=n[e],o=0,a=i.length;a>o;o++)(u=i[o])&(u,o,e);return n}function 
U(n){return sa(n,da),n}function j(n){var t,e;return function(r,u,i){var 
o,a=n[i].update,c=a.length;for(i!=e&&(e=i,t=0),u>=t&&(t=u+1);!(o=a[t])&&++t

[11/16] incubator-airflow git commit: [AIRFLOW-6] Remove dependency on Highcharts

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/hooks/__init__.py
--
diff --git a/airflow/hooks/__init__.py b/airflow/hooks/__init__.py
index 58fac17..cb0aa14 100644
--- a/airflow/hooks/__init__.py
+++ b/airflow/hooks/__init__.py
@@ -1,6 +1,39 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Only import Core Airflow Operators that don't have extra requirements.
+# All other operators must be imported directly.
+from .base_hook import BaseHook
+from .dbapi_hook import DbApiHook
+from .http_hook import HttpHook
+from .sqlite_hook import SqliteHook
+
+# 
+#
+# #TODO #FIXME Airflow 2.0
+#
+# Old import machinary below.
+#
+# This is deprecated but should be kept until Airflow 2.0
+# for compatibility.
+#
+# 
+
 # Imports the hooks dynamically while keeping the package API clean,
 # abstracting the underlying modules
 
+
 from airflow.utils.helpers import import_module_attrs as _import_module_attrs
 from airflow.hooks.base_hook import BaseHook  # noqa to expose in package
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/hooks/dbapi_hook.py
--
diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index 11957ff..ae678aa 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -1,3 +1,16 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 from builtins import str
 from past.builtins import basestring

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/hooks/jdbc_hook.py
--
diff --git a/airflow/hooks/jdbc_hook.py b/airflow/hooks/jdbc_hook.py
index 1f9275f..0b0d47d 100644
--- a/airflow/hooks/jdbc_hook.py
+++ b/airflow/hooks/jdbc_hook.py
@@ -1,11 +1,10 @@
 from builtins import str
-__author__ = 'janomar'
 
-import logging
 import jaydebeapi
 
 from airflow.hooks.dbapi_hook import DbApiHook
 
+
 class JdbcHook(DbApiHook):
 """
 General hook for jdbc db access.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/hooks/oracle_hook.py
--
diff --git a/airflow/hooks/oracle_hook.py b/airflow/hooks/oracle_hook.py
index 6a7f788..ca1ad8f 100644
--- a/airflow/hooks/oracle_hook.py
+++ b/airflow/hooks/oracle_hook.py
@@ -7,6 +7,7 @@ from datetime import datetime
 import numpy
 import logging
 
+
 class OracleHook(DbApiHook):
 """
 Interact with Oracle SQL.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/app.py
--
diff --git a/airflow/www/app.py b/airflow/www/app.py
index 53baf89..b1ef562 100644
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -23,7 +23,7 @@ import airflow
 from airflow import models
 from airflow.settings import Session
 
-from airflow.www.blueprints import ck, routes
+from airflow.www.blueprints import routes
 from airflow import jobs
 from airflow import settings
 from airflow import configuration
@@ -45,9 +45,7 @@ def create_app(config=None):
 cache = Cache(
 app=app, config={'CACHE_TYPE': 'filesystem', 'CACHE_DIR': '/tmp'})
 
-app.register_blueprint(ck, url_prefix='/ck')
 app.register_blueprint(routes)
-app.jinja_env.add_extension("chartkick.ext.charts")
 
 with app.app_context():
 from airflow.www import views

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e606646/airflow/www/blueprints.py
--
diff --git a/airflow/www/blueprints.py 

[02/16] incubator-airflow git commit: Pointing setup.py to then new repo

[bolke]

Pointing setup.py to then new repo


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/4affece9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/4affece9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/4affece9

Branch: refs/heads/branch-1.7.2-apache
Commit: 4affece93c7a0424802d5594d3a804c6a943a884
Parents: 191c21e
Author: Maxime Beauchemin 
Authored: Fri May 20 17:14:21 2016 -0700
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:19:35 2016 +0200

--
 setup.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4affece9/setup.py
--
diff --git a/setup.py b/setup.py
index 2723ed4..905af11 100644
--- a/setup.py
+++ b/setup.py
@@ -187,9 +187,9 @@ def do_setup():
 ],
 author='Maxime Beauchemin',
 author_email='maximebeauche...@gmail.com',
-url='https://github.com/airbnb/airflow',
+url='https://github.com/apache/incubator-airflow',
 download_url=(
-'https://github.com/airbnb/airflow/tarball/' + version),
+'https://github.com/apache/incubator-airflow/tarball/' + version),
 cmdclass={'test': Tox,
   'extra_clean': CleanCommand,
   },

[13/16] incubator-airflow git commit: [AIRFLOW-248] Add Apache license header to all files

[bolke]

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/tests/testconfig/conf/ha-noport-hdfs-site.xml
--
diff --git a/tests/testconfig/conf/ha-noport-hdfs-site.xml 
b/tests/testconfig/conf/ha-noport-hdfs-site.xml
index ae063d9..ec66869 100644
--- a/tests/testconfig/conf/ha-noport-hdfs-site.xml
+++ b/tests/testconfig/conf/ha-noport-hdfs-site.xml
@@ -1,3 +1,19 @@
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/tests/testconfig/conf/ha-noport-trash-hdfs-site.xml
--
diff --git a/tests/testconfig/conf/ha-noport-trash-hdfs-site.xml 
b/tests/testconfig/conf/ha-noport-trash-hdfs-site.xml
index f049558..b5b03a9 100644
--- a/tests/testconfig/conf/ha-noport-trash-hdfs-site.xml
+++ b/tests/testconfig/conf/ha-noport-trash-hdfs-site.xml
@@ -1,3 +1,19 @@
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/tests/testconfig/conf/ha-port-hdfs-site.xml
--
diff --git a/tests/testconfig/conf/ha-port-hdfs-site.xml 
b/tests/testconfig/conf/ha-port-hdfs-site.xml
index 5aa344b..5d758a7 100644
--- a/tests/testconfig/conf/ha-port-hdfs-site.xml
+++ b/tests/testconfig/conf/ha-port-hdfs-site.xml
@@ -1,3 +1,19 @@
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/tests/testconfig/conf/non-ha-port-core-site.xml
--
diff --git a/tests/testconfig/conf/non-ha-port-core-site.xml 
b/tests/testconfig/conf/non-ha-port-core-site.xml
index c686cae..fdfd49f 100644
--- a/tests/testconfig/conf/non-ha-port-core-site.xml
+++ b/tests/testconfig/conf/non-ha-port-core-site.xml
@@ -1,3 +1,19 @@
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/15f710fb/tox.ini
--
diff --git a/tox.ini b/tox.ini
index d86ae2b..3003292 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,3 +1,16 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 [tox]
 envlist = {py27,py34}-{cdh,hdp}-airflow_backend_{mysql,sqlite,postgres}
 skipsdist=True

[16/16] incubator-airflow git commit: Add an Apache Incubator Disclaimer and mocking modules

[bolke]

Add an Apache Incubator Disclaimer and mocking modules

Closes #1634 from mistercrunch/mock_docs

Adding an Apache Incubator Disclaimer and mocking modules


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/27d2d4c7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/27d2d4c7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/27d2d4c7

Branch: refs/heads/branch-1.7.2-apache
Commit: 27d2d4c79c32cb6c19e19e7b9362732a0792aaca
Parents: 15f710f
Author: Maxime Beauchemin 
Authored: Wed Jun 29 13:39:15 2016 -0700
Committer: Bolke de Bruin 
Committed: Mon Jul 11 15:54:30 2016 +0200

--
 airflow/models.py |  4 ++--
 docs/code.rst | 19 ++-
 docs/concepts.rst |  6 ++
 docs/conf.py  | 15 +++
 docs/index.rst| 15 +--
 setup.py  |  7 ---
 6 files changed, 50 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/27d2d4c7/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 3bad273..b60a7a8 100644
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -780,8 +780,8 @@ class TaskInstance(Base):
 Refreshes the task instance from the database based on the primary key
 
 :param lock_for_update: if True, indicates that the database should
-lock the TaskInstance (issuing a FOR UPDATE clause) until the session
-is committed.
+lock the TaskInstance (issuing a FOR UPDATE clause) until the
+session is committed.
 """
 TI = TaskInstance
 

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/27d2d4c7/docs/code.rst
--
diff --git a/docs/code.rst b/docs/code.rst
index 3bec3d7..c693b07 100644
--- a/docs/code.rst
+++ b/docs/code.rst
@@ -89,15 +89,17 @@ Community-contributed Operators
 .. automodule:: airflow.contrib.operators
 :show-inheritance:
 :members:
-BigQueryOperator,
-BigQueryToCloudStorageOperator,
-GoogleCloudStorageDownloadOperator,
-HipChatAPIOperator,
-HipChatAPISendRoomNotificationOperator,
 SSHExecuteOperator,
 VerticaOperator,
 VerticaToHiveTransfer
 
+.. autoclass:: airflow.contrib.operators.bigquery_operator.BigQueryOperator
+.. autoclass:: 
airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator
+.. autoclass:: 
airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator
+.. autoclass:: airflow.contrib.operators.QuboleOperator
+.. autoclass:: airflow.contrib.operators.hipchat_operator.HipChatAPIOperator
+.. autoclass:: 
airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator
+
 .. _macros:
 
 Macros
@@ -223,6 +225,8 @@ Community contributed hooks
 SSHHook,
 CloudantHook
 
+.. autoclass:: airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook
+
 Executors
 -
 Executors are the mechanism by which task instances get run.
@@ -234,7 +238,4 @@ Executors are the mechanism by which task instances get run.
 Community-contributed executors
 '''
 
-.. automodule:: airflow.contrib.executors
-:show-inheritance:
-:members:
-MesosExecutor
+.. autoclass:: airflow.contrib.executors.mesos_executor.MesosExecutor

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/27d2d4c7/docs/concepts.rst
--
diff --git a/docs/concepts.rst b/docs/concepts.rst
index 34e0c33..9a15175 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -664,6 +664,12 @@ configuration files, it allows you to expose the 
configuration that led
 to the related tasks in Airflow.
 
 .. code:: python
+"""
+### My great DAG
+"""
+
+dag = DAG('my_dag', default_args=default_args)
+dag.doc_md = __doc__
 
 t = BashOperator("foo", dag=dag)
 t.doc_md = """\

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/27d2d4c7/docs/conf.py
--
diff --git a/docs/conf.py b/docs/conf.py
index b0b5195..c462847 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -13,6 +13,21 @@
 # serve to show the default.
 import os
 import sys
+import mock
+
+MOCK_MODULES = [
+'apiclient',
+'apiclient.discovery',
+'apiclient.http',
+'mesos',
+'mesos.interface',
+'mesos.native',
+'oauth2client.service_account',
+'pandas.io.gbq',
+]
+for mod_name in MOCK_MODULES:
+sys.modules[mod_name] = mock.Mock()
+
 
 # Hack to allow changing for piece of the code 

[jira] [Created] (AIRFLOW-322) Typo in FAQ section?

[Jimbo Freedman (JIRA)]

Jimbo Freedman created AIRFLOW-322:
--

 Summary: Typo in FAQ section?
 Key: AIRFLOW-322
 URL: https://issues.apache.org/jira/browse/AIRFLOW-322
 Project: Apache Airflow
  Issue Type: Bug
  Components: docs
Affects Versions: Airflow 2.0
Reporter: Jimbo Freedman
Priority: Trivial


I have a feeling that in the third paragraph of "What's the deal with 
``start_date``?", the sentence should read "Airflow will NOW auto-align" 
instead of "Airflow will NOT auto-align" - the context suggests that the 
behaviour has changed. Is this correct?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)