[jira] [Commented] (AIRFLOW-213) Warn if PR commits don't reference the PR

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318541#comment-15318541
 ] 

ASF subversion and git services commented on AIRFLOW-213:
-

Commit cfd502e26068c53cabe6216694e1ae7d35959bc1 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=cfd502e ]

[AIRFLOW-213] Add "Closes #X" phrase to commit messages


> Warn if PR commits don't reference the PR
> -
>
> Key: AIRFLOW-213
> URL: https://issues.apache.org/jira/browse/AIRFLOW-213
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
>
> If a PR is merged but the PR commits don't reference it, GitHub might leave 
> it open (for example AIRFLOW-206). 
> The PR tool should warn if the user deletes "Closes #XXX" from the commit 
> message. 
> Also as part of this resolution I will experiment with an empty commit that 
> contains that message to properly close the PR for AIRFLOW-206. 
> cc [~criccomini]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-206) Support for reading log files locally even if task ran in different host

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318540#comment-15318540
 ] 

ASF subversion and git services commented on AIRFLOW-206:
-

Commit a73d7559cd98c7f6857ee87cb623f9887dda5a18 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a73d755 ]

[AIRFLOW-206] Add commit to close PR

Closes #1566
GitHub requires the above string to properly mark the PR as closed


> Support for reading log files locally even if task ran in different host
> 
>
> Key: AIRFLOW-206
> URL: https://issues.apache.org/jira/browse/AIRFLOW-206
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: webserver
>Affects Versions: Airflow 1.7.1.2
>Reporter: Dante De Nigris
>Assignee: Dante De Nigris
>Priority: Trivial
> Fix For: Airflow 1.8
>
>
> The webserver currently fetches the task processing logs from a http server 
> found on the host that ran that task.
> In the context were the logs are being saved to a directory accessible to 
> both the webserver host and the worker host (e.g. NFS mount) it would be a 
> nice feature to let the webserver read the log directly (i.e. as if it had 
> ran with a LocalExecutor).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-9) Create Airflow website

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318845#comment-15318845
 ] 

ASF subversion and git services commented on AIRFLOW-9:
---

Commit b85fd61d15a85c0cf93e0e5e545ddbbbd710920a in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b85fd61 ]

[AIRFLOW-9] Improving docs to meet Apache's standards


> Create Airflow website
> --
>
> Key: AIRFLOW-9
> URL: https://issues.apache.org/jira/browse/AIRFLOW-9
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: project-management
>Reporter: Chris Riccomini
>
> We should set up an Airflow website for:
> http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-142) setup_env.sh doesn't download hive tarball if hdp is specified as distro

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323388#comment-15323388
 ] 

ASF subversion and git services commented on AIRFLOW-142:
-

Commit b7def7f1f9a97d584e9076cdad48287e652a2d41 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b7def7f ]

[AIRFLOW-142] setup_env.sh doesn't download hive tarball if hdp is specified as 
distro

Closes #1518 from sekikn/AIRFLOW-142


> setup_env.sh doesn't download hive tarball if hdp is specified as distro
> 
>
> Key: AIRFLOW-142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-142
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>
> I tried the following command:
> {code}
> sekikn@ubuntu-trusty:~/incubator-airflow$ env TRAVIS_CACHE=/tmp/cache 
> scripts/ci/setup_env.sh
> {code}
> ... and downloading hive failed as follows:
> {code}
> echo "Downloading and unpacking hive"
> Downloading and unpacking hive
> curl -z ${TRAVIS_CACHE}/hive/hive.tar.gz -o ${TRAVIS_CACHE}/hive/hive.tar.gz 
> -L ${HIVE_URL}
> Warning: Illegal date format for -z, --timecond (and not a file name). 
> Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
> curl: no URL specified!
> curl: try 'curl --help' or 'curl --manual' for more information
> tar zxf ${TRAVIS_CACHE}/hive/hive.tar.gz --strip-components 1 -C ${HIVE_HOME}
> tar (child): /tmp/cache/hive/hive.tar.gz: Cannot open: No such file or 
> directory
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-142) setup_env.sh doesn't download hive tarball if hdp is specified as distro

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323389#comment-15323389
 ] 

ASF subversion and git services commented on AIRFLOW-142:
-

Commit b7def7f1f9a97d584e9076cdad48287e652a2d41 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b7def7f ]

[AIRFLOW-142] setup_env.sh doesn't download hive tarball if hdp is specified as 
distro

Closes #1518 from sekikn/AIRFLOW-142


> setup_env.sh doesn't download hive tarball if hdp is specified as distro
> 
>
> Key: AIRFLOW-142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-142
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>
> I tried the following command:
> {code}
> sekikn@ubuntu-trusty:~/incubator-airflow$ env TRAVIS_CACHE=/tmp/cache 
> scripts/ci/setup_env.sh
> {code}
> ... and downloading hive failed as follows:
> {code}
> echo "Downloading and unpacking hive"
> Downloading and unpacking hive
> curl -z ${TRAVIS_CACHE}/hive/hive.tar.gz -o ${TRAVIS_CACHE}/hive/hive.tar.gz 
> -L ${HIVE_URL}
> Warning: Illegal date format for -z, --timecond (and not a file name). 
> Warning: Disabling time condition. See curl_getdate(3) for valid date syntax.
> curl: no URL specified!
> curl: try 'curl --help' or 'curl --manual' for more information
> tar zxf ${TRAVIS_CACHE}/hive/hive.tar.gz --strip-components 1 -C ${HIVE_HOME}
> tar (child): /tmp/cache/hive/hive.tar.gz: Cannot open: No such file or 
> directory
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-187) Make PR tool more user-friendly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314986#comment-15314986
 ] 

ASF subversion and git services commented on AIRFLOW-187:
-

Commit 2ce8ff565071ff8b69a61ce0e78024305f6bf0d7 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2ce8ff5 ]

[AIRFLOW-187] Improve PR Tool UX

Improved styling for PR tool prompts


> Make PR tool more user-friendly
> ---
>
> Key: AIRFLOW-187
> URL: https://issues.apache.org/jira/browse/AIRFLOW-187
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: PR tool
>Reporter: Jeremiah Lowin
>Priority: Minor
>
> General JIRA improvement that can be referenced for any UX improvements to 
> the PR tool, including better or more prompts, documentation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-157) Minor fixes for PR merge tool

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300240#comment-15300240
 ] 

ASF subversion and git services commented on AIRFLOW-157:
-

Commit 805944b74744b34e1510c2f5d080de98704705d0 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=805944b ]

[AIRFLOW-157] Make PR tool Py3-compat; add JIRA command

- Adds Python3 compatibility (filter objects can't be indexed)
- Adds JIRA command to close issues without merging a PR
- Adds general usability fixes and starts cleaning up code


> Minor fixes for PR merge tool
> -
>
> Key: AIRFLOW-157
> URL: https://issues.apache.org/jira/browse/AIRFLOW-157
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
> Fix For: Airflow 1.8
>
>
> 1. subscripting a {{filter}} object fails in Python3
> 2. JIRA issue inference looks for a 4 or 5 digit issue number... we're not 
> quite there yet!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-170) FileToGoogleCloudStorageOperator fails to use @apply_defaults

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299435#comment-15299435
 ] 

ASF subversion and git services commented on AIRFLOW-170:
-

Commit 85d23a3c4fec6c66b944b196bf2ca69d6c55461d in incubator-airflow's branch 
refs/heads/master from [~firehed]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=85d23a3 ]

[AIRFLOW-170] Add missing @apply_defaults


> FileToGoogleCloudStorageOperator fails to use @apply_defaults
> -
>
> Key: AIRFLOW-170
> URL: https://issues.apache.org/jira/browse/AIRFLOW-170
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: gcp
>Reporter: Eric Stern
>Assignee: Eric Stern
>Priority: Minor
>
> Caught this out of band due to how my connections are configured. PR coming 
> momentarily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-176) PR tool crashes with non-integer JIRA ids

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302156#comment-15302156
 ] 

ASF subversion and git services commented on AIRFLOW-176:
-

Commit beb95a5c67cf25d4110416b3998e89f012ae0489 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=beb95a5 ]

[AIRFLOW-176] Improve PR Tool JIRA workflow

- Fix crash when non-integer IDs are passed
- Improve workflow by always asking user if they
  want to resolve another issue before exiting


> PR tool crashes with non-integer JIRA ids
> -
>
> Key: AIRFLOW-176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-176
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Affects Versions: Airflow 1.7.1.2
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>
> The PR tool crashes if a non-integer id is passed. This includes the default 
> ID  (AIRFLOW-XXX) so it affects folks who don't type in a new ID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305327#comment-15305327
 ] 

ASF subversion and git services commented on AIRFLOW-168:
-

Commit 19fa9852e8ad57036fff12dd28f5b4e67642bce7 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=19fa985 ]

AIRFLOW-168 Correct evaluation of @once schedule

If the schedule @once was used with a start_date two dagruns
would be created as next_run_date would be none and compared
against dag.start_date. This patch fixes that by returning
immediately if a dagrun has already occured with an @once
schedule.


> schedule_interval @once scheduling dag atleast twice
> 
>
> Key: AIRFLOW-168
> URL: https://issues.apache.org/jira/browse/AIRFLOW-168
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.1.2
>Reporter: Sumit Maheshwari
>Assignee: Bolke de Bruin
> Attachments: Screen Shot 2016-05-24 at 9.51.50 PM.png, 
> screenshot-1.png
>
>
> I was looking at example_xcom example and found that it got scheduled twice. 
> Ones at the start_time and ones at the current time. To be correct I tried 
> multiple times (by reloading db) and its same. 
> I am on airflow master, using sequential executor with sqlite3. Though it 
> works as expected on a prod env which is running v1.7 with celery workers and 
> mysql backend.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305328#comment-15305328
 ] 

ASF subversion and git services commented on AIRFLOW-168:
-

Commit e97a1fc759559bb126c4f548b7f71ce8c8c2abe9 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e97a1fc ]

Merge branch 'AIRFLOW-168'


> schedule_interval @once scheduling dag atleast twice
> 
>
> Key: AIRFLOW-168
> URL: https://issues.apache.org/jira/browse/AIRFLOW-168
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.1.2
>Reporter: Sumit Maheshwari
>Assignee: Bolke de Bruin
> Attachments: Screen Shot 2016-05-24 at 9.51.50 PM.png, 
> screenshot-1.png
>
>
> I was looking at example_xcom example and found that it got scheduled twice. 
> Ones at the start_time and ones at the current time. To be correct I tried 
> multiple times (by reloading db) and its same. 
> I am on airflow master, using sequential executor with sqlite3. Though it 
> works as expected on a prod env which is running v1.7 with celery workers and 
> mysql backend.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-68) Align start_date with the schedule_interval

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-68?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325832#comment-15325832
 ] 

ASF subversion and git services commented on AIRFLOW-68:


Commit f69eec3b44cbb0a0fb46a17baec195f7f3baf50e in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f69eec3 ]

[AIRFLOW-68] Align start_date with the schedule_interval

This particular issue arises because of an alignment issue between
start_date and schedule_interval. This can only happen with cron-based
schedule_intervals that describe absolute points in time (like “1am”) as
opposed to time deltas (like “every hour”)

In the past (and in the docs) we have simply said that users must make
sure the two params agree. But this is counter intuitive. As in these
cases, start_date is sort of like telling the scheduler to
“start paying attention” as opposed to “this is my first execution date”.

This patch changes the behavior of the scheduler. The next run date of
the dag will be treated as "start_date + interval" unless the start_date
is on the (previous) interval in which case the start_date will be the
next run date.


> Align start_date with the schedule_interval
> ---
>
> Key: AIRFLOW-68
> URL: https://issues.apache.org/jira/browse/AIRFLOW-68
> Project: Apache Airflow
>  Issue Type: Sub-task
>  Components: scheduler
>Affects Versions: Airflow 1.7.1
>Reporter: Bolke de Bruin
> Fix For: Airflow 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-238) New flask-admin==1.4.1 breaks the Pypi release 1.7.1.2

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329259#comment-15329259
 ] 

ASF subversion and git services commented on AIRFLOW-238:
-

Commit 54b361d2a16a1834234b54292a73d89eeb575228 in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=54b361d ]

[AIRFLOW-238] Make compatible with flask-admin 1.4.1

The new flask-admin==1.4.1 release on 2016-06-13 breaks the Airflow
release currently in Pypi (1.7.1.2). This fixes the edge case triggered
by this new release.

* Closes #1588 on github


> New flask-admin==1.4.1 breaks the Pypi release 1.7.1.2
> --
>
> Key: AIRFLOW-238
> URL: https://issues.apache.org/jira/browse/AIRFLOW-238
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-171) Email does not work in 1.7.1.2

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329266#comment-15329266
 ] 

ASF subversion and git services commented on AIRFLOW-171:
-

Commit 8d501b0ceae39f44f8b96457874f28bb64bfb8c3 in incubator-airflow's branch 
refs/heads/master from Rob Froetscher
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8d501b0 ]

[AIRFLOW-171] Add upgrade notes on email and S3 to 1.7.1.2

Closes #1587 from rfroetscher/upgrading_readme


> Email does not work in 1.7.1.2
> --
>
> Key: AIRFLOW-171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-171
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1
> Environment: AWS Amazon Linux Image
>Reporter: Hao Ye
>Assignee: Hao Ye
>
> Job failure emails was working in 1.7.0. They seem to have stopped working in 
> 1.7.1.
> Error is
> {quote}
> [2016-05-25 00:48:02,334] {models.py:1311} ERROR - Failed to send email to: 
> ['em...@email.com']
> [2016-05-25 00:48:02,334] {models.py:1312} ERROR - 'module' object has no 
> attribute 'send_email_smtp'
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1308, 
> in handle_failure
> self.email_alert(error, is_retry=False)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1425, 
> in email_alert
> send_email(task.email, title, body)
>   File "/usr/local/lib/python2.7/site-packages/airflow/utils/email.py", line 
> 42, in send_email
> backend = getattr(module, attr)
> AttributeError: 'module' object has no attribute 'send_email_smtp'
> {quote}
> File exists and method exists. Seems to work fine when called in python 
> directly.
> Maybe it's loading the wrong email module.
> Tried to set PYTHONPATH to have 
> /usr/local/lib/python2.7/site-packages/airflow earlier in the path, but that 
> didn't seem to work either.
> Could this be related to the utils refactoring that happened between 1.7.0 
> and 1.7.1?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-231) Remove security issue around `eval` statement in PrestoHook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329370#comment-15329370
 ] 

ASF subversion and git services commented on AIRFLOW-231:
-

Commit 7d29698b639d9e2060465aa778efb842986df706 in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7d29698 ]

[AIRFLOW-231] Do not eval user input in PrestoHook

Running `eval` represent a security threat as the interpreter can be
hijacked by the service returning the string getting "evaled", in this
case Presto. It turns out the code I'm changing here was written a long
time ago and misguided, casting a python object to a string and then
evaling it as a useless round trip.

Closes #1584 from mistercrunch/security


> Remove security issue around `eval` statement in PrestoHook
> ---
>
> Key: AIRFLOW-231
> URL: https://issues.apache.org/jira/browse/AIRFLOW-231
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-222) Running task instances should show duration in UI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329461#comment-15329461
 ] 

ASF subversion and git services commented on AIRFLOW-222:
-

Commit 5963ac8b7b7c0b3470ce6171b1cd928485bc9884 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5963ac8 ]

[AIRFLOW-222] Show duration of task instances in ui

At the moment mousing over a task instance in the Airflow tree view
only shows the duration for completed task, and shows "Duration: null"
for running tasks.

Instead the UI should show the current task instance's duration
for running tasks. This patch addresses this.

Closes #1589 from sekikn/AIRFLOW-222


> Running task instances should show duration in UI
> -
>
> Key: AIRFLOW-222
> URL: https://issues.apache.org/jira/browse/AIRFLOW-222
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: ui, webserver
>Reporter: Dan Davydov
>Assignee: Kengo Seki
>Priority: Minor
>  Labels: newbie, starter
>
> At the moment mousing over a task instance (square) in the Airflow tree view 
> only shows the duration for completed task, and shows "Duration: null" for 
> running tasks. Instead the UI should show the current task instance's 
> duration (given by datetime.datetime.now() - ti.start_date) for running tasks.
> Bonus points for incrementing the duration on the client in real-time after 
> it fetches the initial duration. from the server
> Even more bonus points for adding a duration field to the admin/airflow/task 
> endpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-216) Sqoop hook and operator required

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329332#comment-15329332
 ] 

ASF subversion and git services commented on AIRFLOW-216:
-

Commit 717a4aeee3bd0883067c82a3174ad8e2b7dbe8ec in incubator-airflow's branch 
refs/heads/master from [~jwi078]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=717a4ae ]

[AIRFLOW-216] Add Sqoop Hook and Operator

This patch adds a Sqoop hook and operator that implements Sqoop import.
The hook is a wrapper around the sqoop 1 binary.

* Closes #1576 from jwi078/sqoop_operator


> Sqoop hook and operator required
> 
>
> Key: AIRFLOW-216
> URL: https://issues.apache.org/jira/browse/AIRFLOW-216
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, hooks, operators
>Affects Versions: Airflow 1.7.0
>Reporter: Johan
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> It would be nice to have a Sqoop hook and operator for Sqoop 1 which will 
> execute, import, and export Sqoop commands.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-230) [HiveServer2Hook] adding multi statements support

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327923#comment-15327923
 ] 

ASF subversion and git services commented on AIRFLOW-230:
-

Commit a599167c433246d96bea711d8bfd5710b2c9d3ff in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a599167 ]

[AIRFLOW-230] [HiveServer2Hook] adding multi statements support

Changing the library from pyhive to impyla broke the behavior where multiple 
statements, including statements that don't return results were previously 
supported and aren't anymore. impyla raises an exception if any of the 
statements doesn't return result.

We have tasks that run multiple statements including DDL and want to run them 
atomically.

Closes #1583 from mistercrunch/hooks_hive_presto

[AIRFLOW-230] [HiveServer2Hook] adding multi statements support


> [HiveServer2Hook] adding multi statements support
> -
>
> Key: AIRFLOW-230
> URL: https://issues.apache.org/jira/browse/AIRFLOW-230
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-131) XCom data is cleared even if a task already succeeded

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317329#comment-15317329
 ] 

ASF subversion and git services commented on AIRFLOW-131:
-

Commit 18009d03311a0b29e14811865e0b13b19427b5e4 in incubator-airflow's branch 
refs/heads/master from [~john.nason]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=18009d0 ]

[AIRFLOW-131] Make XCom.clear more selective

XCOMs for a task were getting cleared on every run, no matter what.
Selectively clear only when the task is actually going to be run.

Closes #1570 from johnnason/AIRFLOW-131


> XCom data is cleared even if a task already succeeded
> -
>
> Key: AIRFLOW-131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-131
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: xcom
>Affects Versions: Airflow 1.7.1
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>
> In {{TaskInstance.run()}}, the XCom data is immediately cleared. It should 
> not be cleared until we determine that the task is actually going to run. For 
> example, if the task already succeeded it won't run and we want the XCom data 
> to remain.
> Reported here: 
> https://github.com/apache/incubator-airflow/issues/999#issuecomment-220047510



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-131) XCom data is cleared even if a task already succeeded

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317330#comment-15317330
 ] 

ASF subversion and git services commented on AIRFLOW-131:
-

Commit 18009d03311a0b29e14811865e0b13b19427b5e4 in incubator-airflow's branch 
refs/heads/master from [~john.nason]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=18009d0 ]

[AIRFLOW-131] Make XCom.clear more selective

XCOMs for a task were getting cleared on every run, no matter what.
Selectively clear only when the task is actually going to be run.

Closes #1570 from johnnason/AIRFLOW-131


> XCom data is cleared even if a task already succeeded
> -
>
> Key: AIRFLOW-131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-131
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: xcom
>Affects Versions: Airflow 1.7.1
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>
> In {{TaskInstance.run()}}, the XCom data is immediately cleared. It should 
> not be cleared until we determine that the task is actually going to run. For 
> example, if the task already succeeded it won't run and we want the XCom data 
> to remain.
> Reported here: 
> https://github.com/apache/incubator-airflow/issues/999#issuecomment-220047510



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-64) dags_folder not handled correctly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1531#comment-1531
 ] 

ASF subversion and git services commented on AIRFLOW-64:


Commit d32fb8d974f85a316d0cb96d5b6fb2d3502972be in incubator-airflow's branch 
refs/heads/master from [~giacomo@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d32fb8d ]

[AIRFLOW-64] Add note about relative DAGS_FOLDER

Closes #1474 from itajaja/patch-1.


> dags_folder not handled correctly
> -
>
> Key: AIRFLOW-64
> URL: https://issues.apache.org/jira/browse/AIRFLOW-64
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli, executor, scheduler
>Affects Versions: Airflow 1.7.0
> Environment: running on mac os X
> airflow.cfg:
> ```
> airflow_home = .
> dags_folder = ./dags
> ```
>Reporter: Giacomo Tagliabe
>
> running `airflow list_dags` with the command above from a location with a 
> `dags` subdirectory, the cli erros the following message:
> ```
> airflow.utils.AirflowException: subdir has to be part of your DAGS_FOLDER as 
> defined in your airflow.cfg
> ```
> if I change the airflow.cfg to `dags_folder = dags`, list_dags seem to work 
> correctly, but any other task doesn't seem to handle the dags_folder 
> correctly. Here is what `airflow scheduler` outputs when it tries to execute 
> a scheduled dag:
> ```
> DAG [process_deleted_runs] could not be found in 
> /Users/giacomo/workspace/qsi/scheduler/dags/Users/giacomo/workspace/qsi/scheduler/process_deleted_runs.py
> ```
> As you see, it seems that the absolute path 
> (`/Users/giacomo/workspace/qsi/scheduler/`) is chained twice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-25) Configuration for Celery always required

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307316#comment-15307316
 ] 

ASF subversion and git services commented on AIRFLOW-25:


Commit 03e8e5d49f2539057574daecd3f1711e666c29e9 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=03e8e5d ]

[AIRFLOW-25] Configuration for Celery always required

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- AIRFLOW-25

For now, if airflow.cfg has no [celery] section, all subcommands fail.
This patch adds the default values for Celery-related properties
as well as existing 'default_queue' and 'flower_port',
so as to make all subcommands work and suppress "not found in config"
warnings even if [celery] section is omitted.

Author: Kengo Seki 

Closes #1558 from sekikn/AIRFLOW-25.


> Configuration for Celery always required
> 
>
> Key: AIRFLOW-25
> URL: https://issues.apache.org/jira/browse/AIRFLOW-25
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: celery
> Environment: Ubuntu 15.10, Airflow 1.7.0
>Reporter: Ludovic Claude
>Assignee: Kengo Seki
>Priority: Trivial
>  Labels: newbie
>
> I have setup in airflow.cfg
>   airflow_executor = LocalExecutor
> and removed the [celery] section as I do not use it (I want to use 
> MesosExecutor later on)
> When I run
>   airflow initdb
> Airflow fails with this error:
> {code}
> [2016-04-30 23:53:38,430] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-04-30 23:53:38,455] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2016-04-30 23:53:38,583] {configuration.py:424} WARNING - section/key 
> [celery/broker_url] not found in config
> [2016-04-30 23:53:38,583] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-04-30 23:53:38,895] {configuration.py:424} WARNING - section/key 
> [celery/celeryd_concurrency] not found in config
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 13, in 
> parser = get_parser()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 751, 
> in get_parser
> default=configuration.get('celery', 'celeryd_concurrency'))
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 520, in get
> return conf.get(section, key, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/configuration.py", 
> line 428, in get
> "in config".format(**locals()))
> airflow.configuration.AirflowConfigException: section/key 
> [celery/celeryd_concurrency] not found in config
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-23) Support Google cloud DataProc in Airflow

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308110#comment-15308110
 ] 

ASF subversion and git services commented on AIRFLOW-23:


Commit 0a5364cd430a024ed9be4e5b2b5d5174c1b98497 in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0a5364c ]

[AIRFLOW-23] Support for Google Cloud DataProc

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- AIRFLOW-23

Spark, Hadoop, PySpart, SparkSQL, Pig an Hive support of DataProc cluster

Author: Alex Van Boxel 

Closes #1532 from alexvanboxel/AIRFLOW-23.


> Support Google cloud DataProc in Airflow
> 
>
> Key: AIRFLOW-23
> URL: https://issues.apache.org/jira/browse/AIRFLOW-23
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: gcp
>Reporter: Chris Riccomini
>Assignee: Alex Van Boxel
>  Labels: gcp
>
> It would be useful to be able to interact with [Google cloud 
> DataProc|https://cloud.google.com/dataproc/] clusters from Airflow. We should 
> add the proper hooks/operators to do this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-23) Support Google cloud DataProc in Airflow

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308111#comment-15308111
 ] 

ASF subversion and git services commented on AIRFLOW-23:


Commit 0a5364cd430a024ed9be4e5b2b5d5174c1b98497 in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0a5364c ]

[AIRFLOW-23] Support for Google Cloud DataProc

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- AIRFLOW-23

Spark, Hadoop, PySpart, SparkSQL, Pig an Hive support of DataProc cluster

Author: Alex Van Boxel 

Closes #1532 from alexvanboxel/AIRFLOW-23.


> Support Google cloud DataProc in Airflow
> 
>
> Key: AIRFLOW-23
> URL: https://issues.apache.org/jira/browse/AIRFLOW-23
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: gcp
>Reporter: Chris Riccomini
>Assignee: Alex Van Boxel
>  Labels: gcp
>
> It would be useful to be able to interact with [Google cloud 
> DataProc|https://cloud.google.com/dataproc/] clusters from Airflow. We should 
> add the proper hooks/operators to do this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-176) PR tool crashes with non-integer JIRA ids

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302265#comment-15302265
 ] 

ASF subversion and git services commented on AIRFLOW-176:
-

Commit ff7e03bc6b06c9eebb14305272f5cb951a991641 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ff7e03b ]

[AIRFLOW-176] remove unused formatting key


> PR tool crashes with non-integer JIRA ids
> -
>
> Key: AIRFLOW-176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-176
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Affects Versions: Airflow 1.7.1.2
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>
> The PR tool crashes if a non-integer id is passed. This includes the default 
> ID  (AIRFLOW-XXX) so it affects folks who don't type in a new ID.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-167) Get dag state for a given execution date.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302288#comment-15302288
 ] 

ASF subversion and git services commented on AIRFLOW-167:
-

Commit 9db00511da22c731d10cdc4ea40942c77b1b4008 in incubator-airflow's branch 
refs/heads/master from [~sumitm]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=9db0051 ]

AIRFLOW-167: Add dag_state option in cli


> Get dag state for a given execution date.
> -
>
> Key: AIRFLOW-167
> URL: https://issues.apache.org/jira/browse/AIRFLOW-167
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>
> I was trying to get state for a particular dag-run programmatically, but 
> couldn't find a way. 
> If we could have a rest call like 
> `/admin/dagrun?dag_id=_date=` and get the output that 
> would be best. Currently we've to do html parsing to get the same. 
> Other (and easier) way is to add a cli support like we have for `task_state`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-167) Get dag state for a given execution date.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302289#comment-15302289
 ] 

ASF subversion and git services commented on AIRFLOW-167:
-

Commit b9efdc6202e5cb456e55755f2e2ba823fc83a41b in incubator-airflow's branch 
refs/heads/master from [~criccomini]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b9efdc6 ]

Merge pull request #1541 from msumit/AIRFLOW-167


> Get dag state for a given execution date.
> -
>
> Key: AIRFLOW-167
> URL: https://issues.apache.org/jira/browse/AIRFLOW-167
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: cli
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>
> I was trying to get state for a particular dag-run programmatically, but 
> couldn't find a way. 
> If we could have a rest call like 
> `/admin/dagrun?dag_id=_date=` and get the output that 
> would be best. Currently we've to do html parsing to get the same. 
> Other (and easier) way is to add a cli support like we have for `task_state`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-178) Zip files in DAG folder does not get picked up by Ariflow

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302238#comment-15302238
 ] 

ASF subversion and git services commented on AIRFLOW-178:
-

Commit 50f911cd39e9fd9abf3a1d3283cf2e6307ab2540 in incubator-airflow's branch 
refs/heads/master from [~joy.gao54]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=50f911c ]

[AIRFLOW-178] Fix bug so that zip file is detected in DAG folder


> Zip files in DAG folder does not get picked up by Ariflow
> -
>
> Key: AIRFLOW-178
> URL: https://issues.apache.org/jira/browse/AIRFLOW-178
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Joy Gao
>Assignee: Joy Gao
>Priority: Minor
>
> The collect_dags method in DagBag class currently skips any file that does 
> not end in '.py', thereby skipping potential zip files in the DAG folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-183) webserver not retrieving from remote s3/gcs if a log file has been deleted from a remote worker

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304401#comment-15304401
 ] 

ASF subversion and git services commented on AIRFLOW-183:
-

Commit 4aac54d64437ad4aae3020de6debabbc9e911709 in incubator-airflow's branch 
refs/heads/master from [~sayap]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4aac54d ]

[AIRFLOW-183] Fetch log from remote when worker returns 4xx/5xx response

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

- https://issues.apache.org/jira/browse/AIRFLOW-183

This is mainly to make the behavior consistent when some log files have
been deleted from the log folder. Without the change, the remote s3/gcs
fallback will only trigger if the task ran on the local worker.

Author: Yap Sok Ann 

Closes #1551 from sayap/remote-log-remote-worker.


> webserver not retrieving from remote s3/gcs if a log file has been deleted 
> from a remote worker
> ---
>
> Key: AIRFLOW-183
> URL: https://issues.apache.org/jira/browse/AIRFLOW-183
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: Airflow 1.7.1
>Reporter: Yap Sok Ann
>Assignee: Yap Sok Ann
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-183) webserver not retrieving from remote s3/gcs if a log file has been deleted from a remote worker

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304400#comment-15304400
 ] 

ASF subversion and git services commented on AIRFLOW-183:
-

Commit 4aac54d64437ad4aae3020de6debabbc9e911709 in incubator-airflow's branch 
refs/heads/master from [~sayap]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4aac54d ]

[AIRFLOW-183] Fetch log from remote when worker returns 4xx/5xx response

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:

- https://issues.apache.org/jira/browse/AIRFLOW-183

This is mainly to make the behavior consistent when some log files have
been deleted from the log folder. Without the change, the remote s3/gcs
fallback will only trigger if the task ran on the local worker.

Author: Yap Sok Ann 

Closes #1551 from sayap/remote-log-remote-worker.


> webserver not retrieving from remote s3/gcs if a log file has been deleted 
> from a remote worker
> ---
>
> Key: AIRFLOW-183
> URL: https://issues.apache.org/jira/browse/AIRFLOW-183
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: Airflow 1.7.1
>Reporter: Yap Sok Ann
>Assignee: Yap Sok Ann
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-179) DbApiHook string serialization fails when string contains non-ASCII characters

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304585#comment-15304585
 ] 

ASF subversion and git services commented on AIRFLOW-179:
-

Commit 8f63640584ca2dcd15bcd361d1f9a0d995bad315 in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=8f63640 ]

Revert "[AIRFLOW-179] DbApiHook string serialization fails when string contains 
non-ASCII characters"

This reverts commit 87b4b8fa19cb660317198d74f6d51fdde0a7e067.

Reverting as the method used in the dbapi hook is actually package
specific to MySQLdb and would break the sqlite and mssql hooks.


> DbApiHook string serialization fails when string contains non-ASCII characters
> --
>
> Key: AIRFLOW-179
> URL: https://issues.apache.org/jira/browse/AIRFLOW-179
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: John Bodley
>Assignee: John Bodley
> Fix For: Airflow 1.8
>
>
> The DbApiHook.insert_rows(...) method tries to serialize all values to 
> strings using the ASCII codec,  this is problematic if the cell contains 
> non-ASCII characters, i.e.
> >>> from airflow.hooks import DbApiHook
> >>> DbApiHook._serialize_cell('Nguyễn Tấn Dũng')
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 
> 196, in _serialize_cell
> return "'" + str(cell).replace("'", "''") + "'"
>   File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", 
> line 102, in __new__
> return super(newstr, cls).__new__(cls, value)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: 
> ordinal not in range(128)
> Rather than manually trying to serialize and escape values to an ASCII string 
> one should try to serialize the value to string using the character set of 
> the corresponding target database leveraging the connection to mutate the 
> object to the SQL string literal.
> Additionally the escaping logic for single quotes (') within the 
> _serialize_cell method seems wrong, i.e. 
> str(cell).replace("'", "''")
> would escape the string "you're" to be "'you''ve'" as opposed to "'you\'ve'".
> Note an exception should still be thrown if the target encoding is not 
> compatible with the source encoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-179) DbApiHook string serialization fails when string contains non-ASCII characters

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304466#comment-15304466
 ] 

ASF subversion and git services commented on AIRFLOW-179:
-

Commit 87b4b8fa19cb660317198d74f6d51fdde0a7e067 in incubator-airflow's branch 
refs/heads/master from [~john.bod...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=87b4b8f ]

[AIRFLOW-179] DbApiHook string serialization fails when string contains 
non-ASCII characters

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-179

In addition to correctly serializing non-ASCII characters the literal 
transformation also corrects an issue with escaping single quotes (').

Note it was my intention to add another unit test to `test_hive_to_mysql` in 
`tests/core.py` however on inspection the indentations of the various methods 
seemed wrong, methods are nested and it's not apparent what class they refer 
to. Additionally it seems a number of the test cases aren't related to the 
corresponding class.

For testing purposes I simply ran a pipeline which previously failed with the 
following exception,

[2016-05-26 22:03:39,256] {models.py:1286} ERROR - 'ascii' codec can't 
decode byte 0xc3 in position 230: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 
1245, in run
result = task_copy.execute(context=context)
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/operators/hive_to_mysql.py", 
line 88, in execute
mysql.insert_rows(table=self.mysql_table, rows=results)
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 176, 
in insert_rows
l.append(self._serialize_cell(cell))
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 196, 
in _serialize_cell
return "'" + str(cell).replace("'", "''") + "'"
  File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", 
line 102, in __new__
return super(newstr, cls).__new__(cls, value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 230: 
ordinal not in range(128)

and verified with the presence of the fix that the task succeeded and the 
resulting output was correct. Note currently from grokking the code base it 
seems that only `MySqlHook` objects call the the `insert_rows` method.

Author: John Bodley 

Closes #1550 from johnbodley/dbapi_hook_serialization.


> DbApiHook string serialization fails when string contains non-ASCII characters
> --
>
> Key: AIRFLOW-179
> URL: https://issues.apache.org/jira/browse/AIRFLOW-179
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: John Bodley
>Assignee: John Bodley
> Fix For: Airflow 1.8
>
>
> The DbApiHook.insert_rows(...) method tries to serialize all values to 
> strings using the ASCII codec,  this is problematic if the cell contains 
> non-ASCII characters, i.e.
> >>> from airflow.hooks import DbApiHook
> >>> DbApiHook._serialize_cell('Nguyễn Tấn Dũng')
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 
> 196, in _serialize_cell
> return "'" + str(cell).replace("'", "''") + "'"
>   File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", 
> line 102, in __new__
> return super(newstr, cls).__new__(cls, value)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: 
> ordinal not in range(128)
> Rather than manually trying to serialize and escape values to an ASCII string 
> one should try to serialize the value to string using the character set of 
> the corresponding target database leveraging the connection to mutate the 
> object to the SQL string literal.
> Additionally the escaping logic for single quotes (') within the 
> _serialize_cell method seems wrong, i.e. 
> str(cell).replace("'", "''")
> would escape the string "you're" to be "'you''ve'" as opposed to "'you\'ve'".
> Note an exception should still be thrown if the target encoding is not 
> compatible with the source encoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-179) DbApiHook string serialization fails when string contains non-ASCII characters

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304467#comment-15304467
 ] 

ASF subversion and git services commented on AIRFLOW-179:
-

Commit 87b4b8fa19cb660317198d74f6d51fdde0a7e067 in incubator-airflow's branch 
refs/heads/master from [~john.bod...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=87b4b8f ]

[AIRFLOW-179] DbApiHook string serialization fails when string contains 
non-ASCII characters

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-179

In addition to correctly serializing non-ASCII characters the literal 
transformation also corrects an issue with escaping single quotes (').

Note it was my intention to add another unit test to `test_hive_to_mysql` in 
`tests/core.py` however on inspection the indentations of the various methods 
seemed wrong, methods are nested and it's not apparent what class they refer 
to. Additionally it seems a number of the test cases aren't related to the 
corresponding class.

For testing purposes I simply ran a pipeline which previously failed with the 
following exception,

[2016-05-26 22:03:39,256] {models.py:1286} ERROR - 'ascii' codec can't 
decode byte 0xc3 in position 230: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 
1245, in run
result = task_copy.execute(context=context)
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/operators/hive_to_mysql.py", 
line 88, in execute
mysql.insert_rows(table=self.mysql_table, rows=results)
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 176, 
in insert_rows
l.append(self._serialize_cell(cell))
  File 
"/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 196, 
in _serialize_cell
return "'" + str(cell).replace("'", "''") + "'"
  File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", 
line 102, in __new__
return super(newstr, cls).__new__(cls, value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 230: 
ordinal not in range(128)

and verified with the presence of the fix that the task succeeded and the 
resulting output was correct. Note currently from grokking the code base it 
seems that only `MySqlHook` objects call the the `insert_rows` method.

Author: John Bodley 

Closes #1550 from johnbodley/dbapi_hook_serialization.


> DbApiHook string serialization fails when string contains non-ASCII characters
> --
>
> Key: AIRFLOW-179
> URL: https://issues.apache.org/jira/browse/AIRFLOW-179
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: John Bodley
>Assignee: John Bodley
> Fix For: Airflow 1.8
>
>
> The DbApiHook.insert_rows(...) method tries to serialize all values to 
> strings using the ASCII codec,  this is problematic if the cell contains 
> non-ASCII characters, i.e.
> >>> from airflow.hooks import DbApiHook
> >>> DbApiHook._serialize_cell('Nguyễn Tấn Dũng')
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 
> 196, in _serialize_cell
> return "'" + str(cell).replace("'", "''") + "'"
>   File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", 
> line 102, in __new__
> return super(newstr, cls).__new__(cls, value)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: 
> ordinal not in range(128)
> Rather than manually trying to serialize and escape values to an ASCII string 
> one should try to serialize the value to string using the character set of 
> the corresponding target database leveraging the connection to mutate the 
> object to the SQL string literal.
> Additionally the escaping logic for single quotes (') within the 
> _serialize_cell method seems wrong, i.e. 
> str(cell).replace("'", "''")
> would escape the string "you're" to be "'you''ve'" as opposed to "'you\'ve'".
> Note an exception should still be thrown if the target encoding is not 
> compatible with the source encoding.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-196) HttpSensor does not handle exception properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309199#comment-15309199
 ] 

ASF subversion and git services commented on AIRFLOW-196:
-

Commit 92f44f5b5b47fdedf7e5908f24ee28bf242b4634 in incubator-airflow's branch 
refs/heads/master from [~wakem...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=92f44f5 ]

[AIRFLOW-196] Fix bug that exception is not handled in HttpSensor

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- [*AIRFLOW-196*](https://issues.apache.org/jira/browse/AIRFLOW-196)

If exception happens in poke function in HttpSensor, it is not
well handled that make the sensor finish successfully, which is
incorrect obviously.

Author: Junwei Wang 

Closes #1561 from junwei-wang/master.


> HttpSensor does not handle exception properly
> -
>
> Key: AIRFLOW-196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: Airflow 2.0, Airflow 1.8, Airflow 1.7.1, Airflow 1.7.0, 
> Airflow 1.6.2, Airflow 1.7.1.2
>Reporter: Junwei Wang
>Assignee: Junwei Wang
>  Labels: newbie
> Fix For: Airflow 2.0
>
>
> If exception happens in poke function in HttpSensor, it is not
> well handled that make the sensor finish successfully, which is
> incorrect obviously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-196) HttpSensor does not handle exception properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309200#comment-15309200
 ] 

ASF subversion and git services commented on AIRFLOW-196:
-

Commit 92f44f5b5b47fdedf7e5908f24ee28bf242b4634 in incubator-airflow's branch 
refs/heads/master from [~wakem...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=92f44f5 ]

[AIRFLOW-196] Fix bug that exception is not handled in HttpSensor

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- [*AIRFLOW-196*](https://issues.apache.org/jira/browse/AIRFLOW-196)

If exception happens in poke function in HttpSensor, it is not
well handled that make the sensor finish successfully, which is
incorrect obviously.

Author: Junwei Wang 

Closes #1561 from junwei-wang/master.


> HttpSensor does not handle exception properly
> -
>
> Key: AIRFLOW-196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: Airflow 2.0, Airflow 1.8, Airflow 1.7.1, Airflow 1.7.0, 
> Airflow 1.6.2, Airflow 1.7.1.2
>Reporter: Junwei Wang
>Assignee: Junwei Wang
>  Labels: newbie
> Fix For: Airflow 2.0
>
>
> If exception happens in poke function in HttpSensor, it is not
> well handled that make the sensor finish successfully, which is
> incorrect obviously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-196) HttpSensor does not handle exception properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309198#comment-15309198
 ] 

ASF subversion and git services commented on AIRFLOW-196:
-

Commit 92f44f5b5b47fdedf7e5908f24ee28bf242b4634 in incubator-airflow's branch 
refs/heads/master from [~wakem...@gmail.com]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=92f44f5 ]

[AIRFLOW-196] Fix bug that exception is not handled in HttpSensor

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- [*AIRFLOW-196*](https://issues.apache.org/jira/browse/AIRFLOW-196)

If exception happens in poke function in HttpSensor, it is not
well handled that make the sensor finish successfully, which is
incorrect obviously.

Author: Junwei Wang 

Closes #1561 from junwei-wang/master.


> HttpSensor does not handle exception properly
> -
>
> Key: AIRFLOW-196
> URL: https://issues.apache.org/jira/browse/AIRFLOW-196
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: Airflow 2.0, Airflow 1.8, Airflow 1.7.1, Airflow 1.7.0, 
> Airflow 1.6.2, Airflow 1.7.1.2
>Reporter: Junwei Wang
>Assignee: Junwei Wang
>  Labels: newbie
> Fix For: Airflow 2.0
>
>
> If exception happens in poke function in HttpSensor, it is not
> well handled that make the sensor finish successfully, which is
> incorrect obviously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-185) Fix bug when no branches can be discovered automatically

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312624#comment-15312624
 ] 

ASF subversion and git services commented on AIRFLOW-185:
-

Commit c6ae582f83b4aace8e911d9f9e9324fe7e392fac in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=c6ae582 ]

[AIRFLOW-185] Handle empty versions list


> Fix bug when no branches can be discovered automatically
> 
>
> Key: AIRFLOW-185
> URL: https://issues.apache.org/jira/browse/AIRFLOW-185
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: PR tool
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
>
> A function is called on the list of version branches which expects there to 
> be at least one version available. The function call should be skipped if the 
> list is empty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-181) Travis builds fail due to corrupt cache

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302826#comment-15302826
 ] 

ASF subversion and git services commented on AIRFLOW-181:
-

Commit afcd4fcf01696ee26911640cdeb481defd93c3aa in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=afcd4fc ]

AIRFLOW-181 Fix failing unpacking of hadoop by redownloading

curl compares timestamps, but if the file is corrupt this can
result in hadoop tars that are never updated. This adds a retry
without using the cache.


> Travis builds fail due to corrupt cache
> ---
>
> Key: AIRFLOW-181
> URL: https://issues.apache.org/jira/browse/AIRFLOW-181
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> Corrupt cache is preventing from unpacking hadoop. It needs to redownload the 
> distribution without checking the cache



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-155) Documentation for Qubole Operator

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311214#comment-15311214
 ] 

ASF subversion and git services commented on AIRFLOW-155:
-

Commit dce08f68bccac030abe4cf60c5862b6ce7767e38 in incubator-airflow's branch 
refs/heads/master from Sumit Maheshwari
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dce08f6 ]

[AIRFLOW-155] Documentation of Qubole Operator

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- *https://issues.apache.org/jira/browse/AIRFLOW-155*

Thanks,
Sumit

Author: Sumit Maheshwari 

Closes #1560 from msumit/AIRFLOW-155.


> Documentation for Qubole Operator
> -
>
> Key: AIRFLOW-155
> URL: https://issues.apache.org/jira/browse/AIRFLOW-155
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>  Labels: qds
>
> Creating placeholder ticket for documentation work related to Qubole 
> operator.  Please assign this to me.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-201) HiveMetastoreHook kerberos support is broken

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310060#comment-15310060
 ] 

ASF subversion and git services commented on AIRFLOW-201:
-

Commit ea002b7df3bbc06f155874a3db71c1daf4726b6e in incubator-airflow's branch 
refs/heads/master from [~hudbrog]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ea002b7 ]

[AIRFLOW-201] Fix for HiveMetastoreHook + kerberos

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-201

Author: Alexey Ustyantsev 

Closes #1563 from hudbrog/airflow_201.


> HiveMetastoreHook kerberos support is broken
> 
>
> Key: AIRFLOW-201
> URL: https://issues.apache.org/jira/browse/AIRFLOW-201
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: Airflow 1.7.1
>Reporter: Alexey Ustyantsev
>
> Currently, HiveMetastoreHook on 1.7.1 with kerberos enabled fails with errors 
> like:
> {code}
> [2016-05-31 12:25:20,341] {models.py:1286} ERROR - 'sasl.saslwrapper.Client' 
> object is not callable
> Traceback (most recent call last):
>   File 
> "/usr/local/opr/virtualenv_airflow/local/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/models.py",
>  line 1245, in run
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/operators/sensors.py",
>  line 56, in execute
> while not self.poke(context):
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/operators/sensors.py",
>  line 255, in poke
> self.schema, self.table, self.partition)
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/hooks/hive_hooks.py",
>  line 316, in check_for_partition
> self.metastore._oprot.trans.open()
>   File 
> "/usr/local/opr/virtualenv_airflow/local/lib/python2.7/site-packages/thrift_sasl/__init__.py",
>  line 67, in open
> self.sasl = self.sasl_client_factory()
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/hooks/hive_hooks.py",
>  line 293, in sasl_factory
> sasl_client("service", kerberos_service_name)
> TypeError: 'sasl.saslwrapper.Client' object is not callable{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-201) HiveMetastoreHook kerberos support is broken

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310059#comment-15310059
 ] 

ASF subversion and git services commented on AIRFLOW-201:
-

Commit ea002b7df3bbc06f155874a3db71c1daf4726b6e in incubator-airflow's branch 
refs/heads/master from [~hudbrog]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ea002b7 ]

[AIRFLOW-201] Fix for HiveMetastoreHook + kerberos

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-201

Author: Alexey Ustyantsev 

Closes #1563 from hudbrog/airflow_201.


> HiveMetastoreHook kerberos support is broken
> 
>
> Key: AIRFLOW-201
> URL: https://issues.apache.org/jira/browse/AIRFLOW-201
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: Airflow 1.7.1
>Reporter: Alexey Ustyantsev
>
> Currently, HiveMetastoreHook on 1.7.1 with kerberos enabled fails with errors 
> like:
> {code}
> [2016-05-31 12:25:20,341] {models.py:1286} ERROR - 'sasl.saslwrapper.Client' 
> object is not callable
> Traceback (most recent call last):
>   File 
> "/usr/local/opr/virtualenv_airflow/local/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/models.py",
>  line 1245, in run
> result = task_copy.execute(context=context)
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/operators/sensors.py",
>  line 56, in execute
> while not self.poke(context):
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/operators/sensors.py",
>  line 255, in poke
> self.schema, self.table, self.partition)
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/hooks/hive_hooks.py",
>  line 316, in check_for_partition
> self.metastore._oprot.trans.open()
>   File 
> "/usr/local/opr/virtualenv_airflow/local/lib/python2.7/site-packages/thrift_sasl/__init__.py",
>  line 67, in open
> self.sasl = self.sasl_client_factory()
>   File 
> "/usr/local/opr/virtualenv_airflow/lib/python2.7/site-packages/airflow-1.7.1-py2.7.egg/airflow/hooks/hive_hooks.py",
>  line 293, in sasl_factory
> sasl_client("service", kerberos_service_name)
> TypeError: 'sasl.saslwrapper.Client' object is not callable{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-195) Support toggling of SubDag clearing in the CLI for parity with UI functionality

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308974#comment-15308974
 ] 

ASF subversion and git services commented on AIRFLOW-195:
-

Commit 115fe1c6b264c10319470a388c747caaf44a6dc4 in incubator-airflow's branch 
refs/heads/master from [~sanand]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=115fe1c ]

[AIRFLOW-195] : Add toggle support to subdag clearing in the CLI


> Support toggling of SubDag clearing in the CLI for parity with UI 
> functionality
> ---
>
> Key: AIRFLOW-195
> URL: https://issues.apache.org/jira/browse/AIRFLOW-195
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Siddharth Anand
>Assignee: Siddharth Anand
>
> Functionality was added in 
> https://github.com/apache/incubator-airflow/pull/1478 to allow for toggling 
> of subdag clearing via the UI. I'm adding similar toggle functionality to the 
> CLI. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-239) test_hdfs_sensor, etc not collected by nosetests

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332253#comment-15332253
 ] 

ASF subversion and git services commented on AIRFLOW-239:
-

Commit 01952b37b574cbe08a5f2e3987d3360325c63a3e in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=01952b3 ]

[AIRFLOW-239] Fix tests indentation

Closes #1591 from zodiac/deindent-tests


> test_hdfs_sensor, etc not collected by nosetests
> 
>
> Key: AIRFLOW-239
> URL: https://issues.apache.org/jira/browse/AIRFLOW-239
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>
> Due to the indentation at 
> https://github.com/apache/incubator-airflow/blob/5963ac8b7b7c0b3470ce6171b1cd928485bc9884/tests/core.py#L1605-L1623
> The methods test_hdfs_sensor and below are treated as internal functions of 
> test_presto_to_mysql and not collected by nosetests
> ```
> ➜  zodiac-airflow git:(master) export AIRFLOW_RUNALL_TESTS=1
> ➜  zodiac-airflow git:(master) nosetests -v --collect-only 2>&1 | grep 
> test_presto_to_mysql | wc -l
>1
> ➜  zodiac-airflow git:(master) nosetests -v --collect-only 2>&1 | grep 
> test_hdfs_sensor | wc -l
>0
> ➜  zodiac-airflow git:(master)
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-223) Make parametrable the IP on which Flower binds to

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332308#comment-15332308
 ] 

ASF subversion and git services commented on AIRFLOW-223:
-

Commit d31ff09b06cab9b8d095a4a5beb733afcc8bfae6 in incubator-airflow's branch 
refs/heads/master from [~dud]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d31ff09 ]

[AIRFLOW-223] Make parametrable the IP on which Flower binds to


> Make parametrable the IP on which Flower binds to
> -
>
> Key: AIRFLOW-223
> URL: https://issues.apache.org/jira/browse/AIRFLOW-223
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: celery, webserver
>Reporter: dud
>Priority: Minor
>
> Hello
> I'd be glad if Flower could be set to bind to localhost only as could be done 
> for Gunicorn.
> GH PR : https://github.com/apache/incubator-airflow/pull/1582
> dud



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-241) Add testing done section to PR template

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330779#comment-15330779
 ] 

ASF subversion and git services commented on AIRFLOW-241:
-

Commit dffd108ed56c3c16c29fb10ad72205292b7fdcf9 in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dffd108 ]

[AIRFLOW-241] Add testing done section to PR template


> Add testing done section to PR template
> ---
>
> Key: AIRFLOW-241
> URL: https://issues.apache.org/jira/browse/AIRFLOW-241
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-225) Better units for task duration graph

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334545#comment-15334545
 ] 

ASF subversion and git services commented on AIRFLOW-225:
-

Commit 1141165271b0e1546ca231bf8aa412793dc38201 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1141165 ]

[AIRFLOW-225] Better units for task duration graph

Right now the job duration window defaults to hours, which for short lived tasks
results in numbers out to five decimals. This patch adjusts the scale of the 
Y-axis
in accordance with the maximum value of the durations to be shown.


> Better units for task duration graph
> 
>
> Key: AIRFLOW-225
> URL: https://issues.apache.org/jira/browse/AIRFLOW-225
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Reporter: Jakob Homan
>Assignee: Kengo Seki
> Attachments: screenshot-1.png
>
>
> Right now the job duration window defaults to hours, which for short lived 
> tasks results in numbers out to five decimals.  Instead, it should adjust the 
> scale to hours, minutes, seconds, as appropriate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-225) Better units for task duration graph

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334546#comment-15334546
 ] 

ASF subversion and git services commented on AIRFLOW-225:
-

Commit ea2904610651c53853d89e8296ab4c71e0926588 in incubator-airflow's branch 
refs/heads/master from [~criccomini]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ea29046 ]

Merge pull request #1595 from sekikn/AIRFLOW-225


> Better units for task duration graph
> 
>
> Key: AIRFLOW-225
> URL: https://issues.apache.org/jira/browse/AIRFLOW-225
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webserver
>Reporter: Jakob Homan
>Assignee: Kengo Seki
> Attachments: screenshot-1.png
>
>
> Right now the job duration window defaults to hours, which for short lived 
> tasks results in numbers out to five decimals.  Instead, it should adjust the 
> scale to hours, minutes, seconds, as appropriate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-256) test_scheduler_reschedule fails due to heartrate check

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336763#comment-15336763
 ] 

ASF subversion and git services commented on AIRFLOW-256:
-

Commit ab2d71be199708918c1f6d85f0c48c51c777f1e4 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ab2d71b ]

[AIRFLOW-256] Fix test_scheduler_reschedule heartrate

test_scheduler_reschedule runs two schedulerjob quite
fast after one another this sometimes is faster than
the heartrate allows and thus the tasks will not get
rescheduled and the test will fail. Fixed by setting
heartrate to 0.


> test_scheduler_reschedule fails due to heartrate check
> --
>
> Key: AIRFLOW-256
> URL: https://issues.apache.org/jira/browse/AIRFLOW-256
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Bolke de Bruin
>
> test_scheduler_reschedule can fail due to the heartrate check



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-6) Remove high-charts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339493#comment-15339493
 ] 

ASF subversion and git services commented on AIRFLOW-6:
---

Commit 0a460081bc7cba2d05434148f092b87d35aa8cd3 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0a46008 ]

[AIRFLOW-6] Remove dependency on Highcharts

Highcharts' license is not compatible with the Apache 2.0
license. This patch removes Highcharts in favor of d3,
however some charts are not supported anymore.

* This brings Maxime Beauchemin's work to master


> Remove high-charts
> --
>
> Key: AIRFLOW-6
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Chris Riccomini
>Assignee: Maxime Beauchemin
>
> According to LEGAL-246, we can't use High Charts in Airflow. The license is 
> incompatible with Apache. We should replace it with another library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-248) Add Apache license header to all files

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341955#comment-15341955
 ] 

ASF subversion and git services commented on AIRFLOW-248:
-

Commit 3ffa656d97a8e3fcf6e3498ab0a1c882146dc3f5 in incubator-airflow's branch 
refs/heads/master from [~ajayydv]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3ffa656 ]

[AIRFLOW-248] Add Apache license header to all files

- Added Apache license header for files with extension (.service, .in, .mako, 
.properties, .ini, .sh, .ldif, .coveragerc, .cfg, .yml, .conf, .sql, .css, .js, 
.html, .xml.
- Added/Replaced shebang on all .sh files with portable version - 
#!/usr/bin/env bash.
- Skipped third party css and js files. Skipped all minified js files as well.

Closes #1598 from ajayyadava/248


> Add Apache license header to all files
> --
>
> Key: AIRFLOW-248
> URL: https://issues.apache.org/jira/browse/AIRFLOW-248
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Ajay Yadava
>Assignee: Ajay Yadava
> Fix For: Airflow 1.8
>
>
> As part of getting ready for an Apache Release we should add Apache License 
> header to all the files( [mailing list discussion | 
> http://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201606.mbox/%3C43C19AB6-9A7F-4351-862E-540D75490396%40gmail.com%3E]
>  ). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-252) Raise Sqlite exceptions when deleting tasks instance in WebUI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341968#comment-15341968
 ] 

ASF subversion and git services commented on AIRFLOW-252:
-

Commit 79d3be5b10d72538a605c7e24da31ccdd711013c in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=79d3be5 ]

[AIRFLOW-252] Raise Sqlite exceptions when deleting tasks instance in WebUI

If users who use SQLite as backend try to delete a task via browser, it fails 
with an exception.
Though this is a bug on Flask-Admin's side basically, this patch provides a 
workaround for it.


> Raise Sqlite exceptions when deleting tasks instance in WebUI
> -
>
> Key: AIRFLOW-252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-252
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.7.1.3
> Environment: Linux 0f857c51a5ce 4.0.9-boot2docker #1 SMP Thu Sep 10 
> 20:39:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: ryanchou
>Assignee: Kengo Seki
>
> When I try to delete a task instances on browse -> tasks instances page 
> through WebUI. It raise a Sqlite exception
> The airflow version is v1.7.1.3 I haven't seen the version choice in jira 
> issue.
> ```
>  File 
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", 
> line 607, in process
> raise TypeError("SQLite DateTime type only accepts Python "
> StatementError: (exceptions.TypeError) SQLite DateTime type only accepts 
> Python datetime and date objects as input. [SQL: u'SELECT 
> task_instance.task_id AS task_instance_task_id, task_in
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-252) Raise Sqlite exceptions when deleting tasks instance in WebUI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341969#comment-15341969
 ] 

ASF subversion and git services commented on AIRFLOW-252:
-

Commit 072fa8ee53fd10e63dc82aec27cf137f0d5c30d9 in incubator-airflow's branch 
refs/heads/master from [~criccomini]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=072fa8e ]

Merge pull request #1609 from sekikn/AIRFLOW-252


> Raise Sqlite exceptions when deleting tasks instance in WebUI
> -
>
> Key: AIRFLOW-252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-252
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: 1.7.1.3
> Environment: Linux 0f857c51a5ce 4.0.9-boot2docker #1 SMP Thu Sep 10 
> 20:39:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: ryanchou
>Assignee: Kengo Seki
>
> When I try to delete a task instances on browse -> tasks instances page 
> through WebUI. It raise a Sqlite exception
> The airflow version is v1.7.1.3 I haven't seen the version choice in jira 
> issue.
> ```
>  File 
> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", 
> line 607, in process
> raise TypeError("SQLite DateTime type only accepts Python "
> StatementError: (exceptions.TypeError) SQLite DateTime type only accepts 
> Python datetime and date objects as input. [SQL: u'SELECT 
> task_instance.task_id AS task_instance_task_id, task_in
> ```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-162) Allow variables to be exposed in the templates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342254#comment-15342254
 ] 

ASF subversion and git services commented on AIRFLOW-162:
-

Commit 54f1c11b6f46b16bc41c289bab1738fec8ab9608 in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=54f1c11 ]

[AIRFLOW-162] Allow variable to be accessible into templates

Closes #1540 from alexvanboxel/AIRFLOW-162

AIRFLOW-162 Allow variable to be accessible into templates


> Allow variables to be exposed in the templates
> --
>
> Key: AIRFLOW-162
> URL: https://issues.apache.org/jira/browse/AIRFLOW-162
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Trivial
>
> Allow variables to be exposed in the templates. This makes it possible to 
> access them in the following way, example:
> {var.gcp_dataflow_base}/test-pipleline.jar
> In this example the basepath in configured in variables. This makes it 
> possible to make some parts configurable (example for differences in 
> prod/staging/test).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-162) Allow variables to be exposed in the templates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342255#comment-15342255
 ] 

ASF subversion and git services commented on AIRFLOW-162:
-

Commit 54f1c11b6f46b16bc41c289bab1738fec8ab9608 in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=54f1c11 ]

[AIRFLOW-162] Allow variable to be accessible into templates

Closes #1540 from alexvanboxel/AIRFLOW-162

AIRFLOW-162 Allow variable to be accessible into templates


> Allow variables to be exposed in the templates
> --
>
> Key: AIRFLOW-162
> URL: https://issues.apache.org/jira/browse/AIRFLOW-162
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Trivial
>
> Allow variables to be exposed in the templates. This makes it possible to 
> access them in the following way, example:
> {var.gcp_dataflow_base}/test-pipleline.jar
> In this example the basepath in configured in variables. This makes it 
> possible to make some parts configurable (example for differences in 
> prod/staging/test).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-162) Allow variables to be exposed in the templates

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342256#comment-15342256
 ] 

ASF subversion and git services commented on AIRFLOW-162:
-

Commit 54f1c11b6f46b16bc41c289bab1738fec8ab9608 in incubator-airflow's branch 
refs/heads/master from [~alexvanboxel]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=54f1c11 ]

[AIRFLOW-162] Allow variable to be accessible into templates

Closes #1540 from alexvanboxel/AIRFLOW-162

AIRFLOW-162 Allow variable to be accessible into templates


> Allow variables to be exposed in the templates
> --
>
> Key: AIRFLOW-162
> URL: https://issues.apache.org/jira/browse/AIRFLOW-162
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Trivial
>
> Allow variables to be exposed in the templates. This makes it possible to 
> access them in the following way, example:
> {var.gcp_dataflow_base}/test-pipleline.jar
> In this example the basepath in configured in variables. This makes it 
> possible to make some parts configurable (example for differences in 
> prod/staging/test).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-244) Expose task/dag id/run data for ingestion by performance monitoring tools

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342448#comment-15342448
 ] 

ASF subversion and git services commented on AIRFLOW-244:
-

Commit 7f6d877409175cf5b0049406d808030d58dcdb24 in incubator-airflow's branch 
refs/heads/master from [~pbramsen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7f6d877 ]

[AIRFLOW-244] Modify hive operator to inject analysis data

Testing Done:
Test dags were run as backfills on an Airbnb Airflow dev box.

This PR exposes task/dag id/run data through the HiveOperator for
ingestion by performance analysis tools like Dr. Elephant.

Closes #1607 from 
paulbramsen/paulbramsen/modify_hive_operator_to_inject_analysis_data


> Expose task/dag id/run data for ingestion by performance monitoring tools
> -
>
> Key: AIRFLOW-244
> URL: https://issues.apache.org/jira/browse/AIRFLOW-244
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Paul Bramsen
>Assignee: Paul Bramsen
>
> Expose task/dag id/run data for ingestion by performance analysis tools like 
> Dr. Elephant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-234) make task that aren't `running` self-terminate

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336999#comment-15336999
 ] 

ASF subversion and git services commented on AIRFLOW-234:
-

Commit 7c0f8373f59b0554d5ba15bb0e5e8669f0830313 in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7c0f837 ]

[AIRFLOW-234] make task that aren't `running` self-terminate

Closes #1585 from mistercrunch/undeads


> make task that aren't `running` self-terminate
> --
>
> Key: AIRFLOW-234
> URL: https://issues.apache.org/jira/browse/AIRFLOW-234
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-257) airflow command fails with "ImportError: No module named zope.deprecation"

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339871#comment-15339871
 ] 

ASF subversion and git services commented on AIRFLOW-257:
-

Commit 45b735baeac794b54dd89ced2f43eec54adf13f7 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=45b735b ]

[AIRFLOW-31] Add zope dependency

Closes #1608 from jlowin/standard-imports-2.
Also closes AIRFLOW-257.


> airflow command fails with "ImportError: No module named zope.deprecation"
> --
>
> Key: AIRFLOW-257
> URL: https://issues.apache.org/jira/browse/AIRFLOW-257
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Kengo Seki
>Priority: Critical
>
> After AIRFLOW-31 has been merged, airflow command fails as follows:
> {code}
> $ airflow webserver
> [2016-06-18 00:56:50,367] {__init__.py:36} INFO - Using executor 
> SequentialExecutor
> [2016-06-18 00:56:50,492] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2016-06-18 00:56:50,529] {driver.py:120} INFO - Generating grammar tables 
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Traceback (most recent call last):
>   File "/home/sekikn/.virtualenvs/e/bin/airflow", line 6, in 
> exec(compile(open(__file__).read(), __file__, 'exec'))
>   File "/home/sekikn/dev/incubator-airflow/airflow/bin/airflow", line 4, in 
> 
> from airflow import configuration
>   File "/home/sekikn/dev/incubator-airflow/airflow/__init__.py", line 76, in 
> 
> from airflow import operators
>   File "/home/sekikn/dev/incubator-airflow/airflow/operators/__init__.py", 
> line 24, in 
> from .check_operator import (
>   File 
> "/home/sekikn/dev/incubator-airflow/airflow/operators/check_operator.py", 
> line 20, in 
> from airflow.hooks import BaseHook
>   File "/home/sekikn/dev/incubator-airflow/airflow/hooks/__init__.py", line 
> 66, in 
> from zope.deprecation import deprecated as _deprecated
> ImportError: No module named zope.deprecation
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-31) Use standard imports for hooks/operators

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339870#comment-15339870
 ] 

ASF subversion and git services commented on AIRFLOW-31:


Commit 45b735baeac794b54dd89ced2f43eec54adf13f7 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=45b735b ]

[AIRFLOW-31] Add zope dependency

Closes #1608 from jlowin/standard-imports-2.
Also closes AIRFLOW-257.


> Use standard imports for hooks/operators
> 
>
> Key: AIRFLOW-31
> URL: https://issues.apache.org/jira/browse/AIRFLOW-31
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>  Labels: enhancement
> Fix For: Airflow 1.8
>
>
> (Migrated from https://github.com/airbnb/airflow/issues/1238)
> Currently, Airflow uses a relatively complex import mechanism to import hooks 
> and operators without polluting the namespace with submodules. I would like 
> to propose that Airflow abandon that system and use standard Python importing.
> Here are a few major reasons why I think the current system has run its 
> course.
> h3. Polluting namespace
> The biggest advantage of the current system, as I understand it, is that only 
> Operators appear in the `airflow.operators` namespace.  The submodules that 
> actually contain the operators do not.
> So for example while `airflow.operators.python_operator.PythonOperator` is a 
> thing, `PythonOperator` is in the `airflow.operators` namespace but 
> `python_operator` is not.
> I think this sort of namespace pollution was helpful when Airflow was a 
> smaller project, but as the number of hooks/operators grows -- and especially 
> as the `contrib` hooks/operators grow -- I'd argue that namespacing is a 
> *good thing*. It provides structure and organization, and opportunities for 
> documentation (through module docstrings).
> In fact, I'd argue that the current namespace is itself getting quite 
> polluted -- the only way to know what's available is to use something like 
> Ipython tab-completion to browse an alphabetical list of Operator names, or 
> to load the source file and grok the import definition (which no one 
> installing from pypi is likely to do).
> h3. Conditional imports
> There's a second advantage to the current system that any module that fails 
> to import is silently ignored. It makes it easy to have optional 
> dependencies. For example, if someone doesn't have `boto` installed, then 
> they don't have an `S3Hook` either. Same for a HiveOperator
> Again, as Airflow grows and matures, I think this is a little too magic. If 
> my environment is missing a dependency, I want to hear about it.
> On the other hand, the `contrib` namespace sort of depends on this -- we 
> don't want users to have to install every single dependency. So I propose 
> that contrib modules all live in their submodules: `from 
> airflow.contrib.operators.my_operator import MyOperator`. As mentioned 
> previously, having structure and namespacing is a good thing as the project 
> gets more complex.
> Other ways to handle this include putting "non-standard" dependencies inside 
> the operator/hook rather than the module (see `HiveOperator`/`HiveHook`), so 
> it can be imported but not used. Another is judicious use of `try`/`except 
> ImportError`. The simplest is to make people import things explicitly from 
> submodules.
> h3. Operator dependencies
> Right now, operators can't depend on each other if they aren't in the same 
> file. This is for the simple reason that there is no guarantee on what order 
> the operators will be loaded. It all comes down to which dictionary key gets 
> loaded first. One day Operator B could be loaded after Operator A; the next 
> day it might be loaded before. Consequently, A and B can't depend on each 
> other. Worse, if a user makes two operators that do depend on each other, 
> they won't get an error message when one fails to import.
> For contrib modules in particular, this is sort of killer.
> h3. Ease of use
> It's *hard* to set up imports for a new operator. The dictionary-based import 
> instructions aren't obvious for new users, and errors are silently dismissed 
> which makes debugging difficult.
> h3. Identity
> Surprisingly, `airflow.operators.SubDagOperator != 
> airflow.operators.subdag_operator.SubDagOperator`. See #1168.
> h2. Proposal
> Use standard python importing for hooks/operators/etc.
> - `__init__.py` files use straightforward, standard Python imports
> - major operators are available at `airflow.operators.OperatorName` or 
> `airflow.operators.operator_module.OperatorName`.
> - contrib operators are only available at 
> 

[jira] [Commented] (AIRFLOW-263) Backtick file introduced by Highcharts refactor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341235#comment-15341235
 ] 

ASF subversion and git services commented on AIRFLOW-263:
-

Commit f4c44146cd38afe21c517230bff71d33ab041246 in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f4c4414 ]

[AIRFLOW-263] Remove temp backtick file

Closes #1613 from artwr/artwr_remove_backtick_file

[AIRFLOW-263] Remove temp backtick file

This removes the file erroneously introduced by the highcharts
refactor.

Solves AIRFLOW-263


> Backtick file introduced by Highcharts refactor
> ---
>
> Key: AIRFLOW-263
> URL: https://issues.apache.org/jira/browse/AIRFLOW-263
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>
> A file named "`" was introduced during the Highcharts removal. See
> https://github.com/apache/incubator-airflow/commit/0a460081bc7cba2d05434148f092b87d35aa8cd3
> My best assessment, is that this was a temporary file created by mistake.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-263) Backtick file introduced by Highcharts refactor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341236#comment-15341236
 ] 

ASF subversion and git services commented on AIRFLOW-263:
-

Commit f4c44146cd38afe21c517230bff71d33ab041246 in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f4c4414 ]

[AIRFLOW-263] Remove temp backtick file

Closes #1613 from artwr/artwr_remove_backtick_file

[AIRFLOW-263] Remove temp backtick file

This removes the file erroneously introduced by the highcharts
refactor.

Solves AIRFLOW-263


> Backtick file introduced by Highcharts refactor
> ---
>
> Key: AIRFLOW-263
> URL: https://issues.apache.org/jira/browse/AIRFLOW-263
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>
> A file named "`" was introduced during the Highcharts removal. See
> https://github.com/apache/incubator-airflow/commit/0a460081bc7cba2d05434148f092b87d35aa8cd3
> My best assessment, is that this was a temporary file created by mistake.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-263) Backtick file introduced by Highcharts refactor

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341234#comment-15341234
 ] 

ASF subversion and git services commented on AIRFLOW-263:
-

Commit f4c44146cd38afe21c517230bff71d33ab041246 in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f4c4414 ]

[AIRFLOW-263] Remove temp backtick file

Closes #1613 from artwr/artwr_remove_backtick_file

[AIRFLOW-263] Remove temp backtick file

This removes the file erroneously introduced by the highcharts
refactor.

Solves AIRFLOW-263


> Backtick file introduced by Highcharts refactor
> ---
>
> Key: AIRFLOW-263
> URL: https://issues.apache.org/jira/browse/AIRFLOW-263
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>
> A file named "`" was introduced during the Highcharts removal. See
> https://github.com/apache/incubator-airflow/commit/0a460081bc7cba2d05434148f092b87d35aa8cd3
> My best assessment, is that this was a temporary file created by mistake.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-273) Vectorized Logos

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347075#comment-15347075
 ] 

ASF subversion and git services commented on AIRFLOW-273:
-

Commit bd9127ebb3149aae7e845b1c4cce60e42d1f677a in incubator-airflow's branch 
refs/heads/master from [~gwax]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=bd9127e ]

[AIRFLOW-273] Create an svg version of the airflow logo.

Closes #1619 from gwax/svg_logo


> Vectorized Logos
> 
>
> Key: AIRFLOW-273
> URL: https://issues.apache.org/jira/browse/AIRFLOW-273
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: George Leslie-Waksman
>Assignee: George Leslie-Waksman
>Priority: Trivial
>
> There has been interest on the mailing list in a SVG version of the logo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-274) Add XCom functionality to GoogleCloudStorageDownloadOperator

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347624#comment-15347624
 ] 

ASF subversion and git services commented on AIRFLOW-274:
-

Commit 518e0073a81585b39757afb4c0d8d65d400a88e2 in incubator-airflow's branch 
refs/heads/master from [~illop]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=518e007 ]

[AIRFLOW-274] Add XCom functionality to GoogleCloudStorageDownloadOperator

Updated GoogleCloudStorageDownloadOperator so that it has the option to
store the downloaded file's contents in an XCom instead of saving to
disk. It now also takes filename as an optional argument instead of a
required argument, so it is not necessary to save to disk if you do not
need to.

Closes #1618 from illop/risk_analytics


> Add XCom functionality to GoogleCloudStorageDownloadOperator
> 
>
> Key: AIRFLOW-274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-274
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Ilya Rakoshes
>Priority: Minor
>
> GoogleCloudStorageDownloadOperator needs to be updated to be able to store 
> the downloaded file in an XCom if the user wants to pass the data to 
> downstream tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-278) Support utf-8 ecoding for sql query strings

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351224#comment-15351224
 ] 

ASF subversion and git services commented on AIRFLOW-278:
-

Commit 5db7ec70f8b45bc4872ae64901d2829efa0e4081 in incubator-airflow's branch 
refs/heads/master from [~bilevn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5db7ec7 ]

[AIRFLOW-278] Support utf-8 ecoding for SQL

Support utf-8 encoding for SQL queries, needed for Python 2
users who have unicode strings inside the queries

Closes #1622 from biln/master


> Support utf-8 ecoding for sql query strings
> ---
>
> Key: AIRFLOW-278
> URL: https://issues.apache.org/jira/browse/AIRFLOW-278
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Nikolay Bilev
>Assignee: Nikolay Bilev
>
> 'utf-8' encoding for python 2 users who has unicode strings inside of sql 
> queries. Default 'ancii' codec doesn't handle them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-280) clean up the tmp druid table no matter if an ingestion job succeeds or not

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349735#comment-15349735
 ] 

ASF subversion and git services commented on AIRFLOW-280:
-

Commit 3b84bcb3eef629da69d10db8a99418afd0386193 in incubator-airflow's branch 
refs/heads/master from [~hzeng]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3b84bcb ]

[AIRFLOW-280] clean up tmp druid table no matter if an ingestion job succeeds 
or not

Closes #1624 from hongbozeng/cleanup_druid

clean up tmp druid table no matter if the ingestion job success or not


> clean up the tmp druid table no matter if an ingestion job succeeds or not
> --
>
> Key: AIRFLOW-280
> URL: https://issues.apache.org/jira/browse/AIRFLOW-280
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: Airflow 1.8
>Reporter: Hongbo Zeng
>Assignee: Hongbo Zeng
> Fix For: Airflow 1.8
>
>
> The tmp table for Druid data will stay if a batch ingestion fails, we need to 
> clean up the hive table no matter an ingestion job succeeds or not



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-31) Use standard imports for hooks/operators

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-31?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335919#comment-15335919
 ] 

ASF subversion and git services commented on AIRFLOW-31:


Commit 851adc5547597ec51743be4bc47d634c77d6dc17 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=851adc5 ]

[AIRFLOW-31] Use standard imports for hooks/operators


> Use standard imports for hooks/operators
> 
>
> Key: AIRFLOW-31
> URL: https://issues.apache.org/jira/browse/AIRFLOW-31
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>  Labels: enhancement
> Fix For: Airflow 2.0
>
>
> (Migrated from https://github.com/airbnb/airflow/issues/1238)
> Currently, Airflow uses a relatively complex import mechanism to import hooks 
> and operators without polluting the namespace with submodules. I would like 
> to propose that Airflow abandon that system and use standard Python importing.
> Here are a few major reasons why I think the current system has run its 
> course.
> h3. Polluting namespace
> The biggest advantage of the current system, as I understand it, is that only 
> Operators appear in the `airflow.operators` namespace.  The submodules that 
> actually contain the operators do not.
> So for example while `airflow.operators.python_operator.PythonOperator` is a 
> thing, `PythonOperator` is in the `airflow.operators` namespace but 
> `python_operator` is not.
> I think this sort of namespace pollution was helpful when Airflow was a 
> smaller project, but as the number of hooks/operators grows -- and especially 
> as the `contrib` hooks/operators grow -- I'd argue that namespacing is a 
> *good thing*. It provides structure and organization, and opportunities for 
> documentation (through module docstrings).
> In fact, I'd argue that the current namespace is itself getting quite 
> polluted -- the only way to know what's available is to use something like 
> Ipython tab-completion to browse an alphabetical list of Operator names, or 
> to load the source file and grok the import definition (which no one 
> installing from pypi is likely to do).
> h3. Conditional imports
> There's a second advantage to the current system that any module that fails 
> to import is silently ignored. It makes it easy to have optional 
> dependencies. For example, if someone doesn't have `boto` installed, then 
> they don't have an `S3Hook` either. Same for a HiveOperator
> Again, as Airflow grows and matures, I think this is a little too magic. If 
> my environment is missing a dependency, I want to hear about it.
> On the other hand, the `contrib` namespace sort of depends on this -- we 
> don't want users to have to install every single dependency. So I propose 
> that contrib modules all live in their submodules: `from 
> airflow.contrib.operators.my_operator import MyOperator`. As mentioned 
> previously, having structure and namespacing is a good thing as the project 
> gets more complex.
> Other ways to handle this include putting "non-standard" dependencies inside 
> the operator/hook rather than the module (see `HiveOperator`/`HiveHook`), so 
> it can be imported but not used. Another is judicious use of `try`/`except 
> ImportError`. The simplest is to make people import things explicitly from 
> submodules.
> h3. Operator dependencies
> Right now, operators can't depend on each other if they aren't in the same 
> file. This is for the simple reason that there is no guarantee on what order 
> the operators will be loaded. It all comes down to which dictionary key gets 
> loaded first. One day Operator B could be loaded after Operator A; the next 
> day it might be loaded before. Consequently, A and B can't depend on each 
> other. Worse, if a user makes two operators that do depend on each other, 
> they won't get an error message when one fails to import.
> For contrib modules in particular, this is sort of killer.
> h3. Ease of use
> It's *hard* to set up imports for a new operator. The dictionary-based import 
> instructions aren't obvious for new users, and errors are silently dismissed 
> which makes debugging difficult.
> h3. Identity
> Surprisingly, `airflow.operators.SubDagOperator != 
> airflow.operators.subdag_operator.SubDagOperator`. See #1168.
> h2. Proposal
> Use standard python importing for hooks/operators/etc.
> - `__init__.py` files use straightforward, standard Python imports
> - major operators are available at `airflow.operators.OperatorName` or 
> `airflow.operators.operator_module.OperatorName`.
> - contrib operators are only available at 
> `airflow.contrib.operators.operator_module.OperatorName` in order to manage 
> 

[jira] [Commented] (AIRFLOW-189) Highlighting of Parent/Child nodes in Graphs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354843#comment-15354843
 ] 

ASF subversion and git services commented on AIRFLOW-189:
-

Commit 0923356a5e18f9767aceccaa02c29144552bf849 in incubator-airflow's branch 
refs/heads/master from withnale
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0923356 ]

[AIRFLOW-189] Highlighting of Parent/Child nodes in Graphs

Closes #1554 from withnale/graph_highlight


> Highlighting of Parent/Child nodes in Graphs
> 
>
> Key: AIRFLOW-189
> URL: https://issues.apache.org/jira/browse/AIRFLOW-189
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Paul Rhodes
>Assignee: Paul Rhodes
>Priority: Minor
> Attachments: Screenshot from 2016-05-29 15-19-39.png
>
>
> Sometimes in large graphs it's difficult to see the direct 
> ancestors/descendants. This change would use mouseover events to highlight 
> these tasks to make it easier to interpret the graphs.
> An example can be seen here in the attachment
> !Screenshot from 2016-05-29 15-19-39.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-150) setup.py classifiers should be a list not a dict

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293798#comment-15293798
 ] 

ASF subversion and git services commented on AIRFLOW-150:
-

Commit af1164059234a88173a23e0cea8a59e757a3f112 in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=af11640 ]

[AIRFLOW-150] setup.py classifiers dict should be list


> setup.py classifiers should be a list not a dict
> 
>
> Key: AIRFLOW-150
> URL: https://issues.apache.org/jira/browse/AIRFLOW-150
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Dan Davydov
>Assignee: Chris Riccomini
>  Labels: newbie
>
> I ran into this issue when trying to release to pypi, setuptools expects 
> classifiers in setup.py to be a list, not a dict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-134) Improve GitHub PR workflow

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293943#comment-15293943
 ] 

ASF subversion and git services commented on AIRFLOW-134:
-

Commit 7e56bd4f9491b16cfafc48c888375fc856730d69 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7e56bd4 ]

[AIRFLOW-134] Add PR merge script


> Improve GitHub PR workflow
> --
>
> Key: AIRFLOW-134
> URL: https://issues.apache.org/jira/browse/AIRFLOW-134
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>
> Merging PRs is hard to do and easy to mess up. Make it better!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-125) Add operator to upload a file to Google Cloud Storage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290021#comment-15290021
 ] 

ASF subversion and git services commented on AIRFLOW-125:
-

Commit 4b25a7d34ea17c86c4b40a09f85898ea4769b22b in incubator-airflow's branch 
refs/heads/master from [~firehed]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4b25a7d ]

[AIRFLOW-125] Add file to GCS operator

Adds an operator to upload a file to Google Cloud Storage. Used as follows:
```py
from airflow.contrib.operators.file_to_gcs import 
FileToGoogleCloudStorageOperator
gcs = FileToGoogleCloudStorageOperator(
bucket='a-bucket-i-have-access-to-on-gcs',
dag=dag,
task_id='upload_stuff',
google_cloud_storage_conn_id='an-airflow-bigquery-connection',
src=os.path.join(os.path.dirname(__file__), 'csv/some_file.csv'),
dst='project/some_file.csv')
```


> Add operator to upload a file to Google Cloud Storage
> -
>
> Key: AIRFLOW-125
> URL: https://issues.apache.org/jira/browse/AIRFLOW-125
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, gcp
>Affects Versions: Airflow 1.7.1
>Reporter: Eric Stern
>Assignee: Eric Stern
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-127) Makes filter_by_owner aware of multi-owner DAG

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290336#comment-15290336
 ] 

ASF subversion and git services commented on AIRFLOW-127:
-

Commit 843a22f74c01612b1858c69b86d0eb5b1cc89a38 in incubator-airflow's branch 
refs/heads/master from [~dud]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=843a22f ]

[AIRFLOW-127] Makes filter_by_owner aware of multi-owner DAG


> Makes filter_by_owner aware of multi-owner DAG
> --
>
> Key: AIRFLOW-127
> URL: https://issues.apache.org/jira/browse/AIRFLOW-127
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: security, webserver
>Reporter: dud
>Priority: Minor
>
> Hello
> When enabling the _webserver.filter_by_owner_ setting, Airflow issues a 
> request like this one :
> {code}
> SELECT * FROM dag WHERE NOT dag.is_subdag AND dag.is_active AND dag.owners = 
> username
> {code}
> Unfortunately that doesn't account for DAG authored by multiple users.
> Please see my PR : https://github.com/apache/incubator-airflow/pull/1508
> dud



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-143) setup_env.sh doesn't leverage cache for downloading minicluster

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291489#comment-15291489
 ] 

ASF subversion and git services commented on AIRFLOW-143:
-

Commit 4b78e1a0f1e9d24c182e8bb04216043c44d3ea76 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4b78e1a ]

[AIRFLOW-143] setup_env.sh doesn't leverage cache for downloading minicluster


> setup_env.sh doesn't leverage cache for downloading minicluster
> ---
>
> Key: AIRFLOW-143
> URL: https://issues.apache.org/jira/browse/AIRFLOW-143
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: tests
>Reporter: Kengo Seki
>Priority: Minor
>
> In scripts/ci/setup_env.sh:
> {code}
> 105 echo "Downloading and unpacking minicluster"
> 106 rm -rf ${TRAVIS_CACHE}/minicluster/minicluster.zip
> 107 curl -z ${TRAVIS_CACHE}/minicluster/minicluster.zip -o 
> ${TRAVIS_CACHE}/minicluster/minicluster.zip -L ${MINICLUSTER_URL}
> 108 unzip ${TRAVIS_CACHE}/minicluster/minicluster.zip -d /tmp
> {code}
> -z option is specified at line 107, but it doesn't seem to make sense because 
> the cache is removed at line 106.
> I think removing cache is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-153) Minor typos in the help message

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294155#comment-15294155
 ] 

ASF subversion and git services commented on AIRFLOW-153:
-

Commit defb0c05b6cc72b950c9793e106a4ef511ab9b80 in incubator-airflow's branch 
refs/heads/master from [~MarkReddy]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=defb0c0 ]

[AIRFLOW-153] fixed minor typos in help message


> Minor typos in the help message
> ---
>
> Key: AIRFLOW-153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-153
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli, docs
>Reporter: Kengo Seki
>Priority: Trivial
>  Labels: newbie
>
> {code}
> $ airflow -h
> (snip)
> unpause Pause a DAG
> {code}
> "Resume a DAG"?
> {code}
> scheduler   Start a scheduler scheduler instance
> {code}
> "Start a scheduler instance"?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-153) Minor typos in the help message

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294156#comment-15294156
 ] 

ASF subversion and git services commented on AIRFLOW-153:
-

Commit e711028acfd0da151498105d8224e0ccdaaf058a in incubator-airflow's branch 
refs/heads/master from [~criccomini]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e711028 ]

Merge pull request #1533 from markreddy/AIRFLOW-153-fix-typos


> Minor typos in the help message
> ---
>
> Key: AIRFLOW-153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-153
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: cli, docs
>Reporter: Kengo Seki
>Priority: Trivial
>  Labels: newbie
>
> {code}
> $ airflow -h
> (snip)
> unpause Pause a DAG
> {code}
> "Resume a DAG"?
> {code}
> scheduler   Start a scheduler scheduler instance
> {code}
> "Start a scheduler instance"?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291680#comment-15291680
 ] 

ASF subversion and git services commented on AIRFLOW-52:


Commit 1d0d8681ddf89fabd5f3521d0846a758e47962b3 in incubator-airflow's branch 
refs/heads/airbnb_rb1.7.1_3 from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1d0d868 ]

[AIRFLOW-52] 1.7.1 version bump and changelog


> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
>  Issue Type: Task
>  Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>  Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added 
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a 
> given non-trivial DAG parses under X seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291677#comment-15291677
 ] 

ASF subversion and git services commented on AIRFLOW-52:


Commit 0b3d101ff5aa430ff4e9e709523192fc6e6c6dee in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0b3d101 ]

[AIRFLOW-52] 1.7.1 version bump and changelog


> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
>  Issue Type: Task
>  Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>  Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added 
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a 
> given non-trivial DAG parses under X seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-121) Documenting dag doc_md feature

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286814#comment-15286814
 ] 

ASF subversion and git services commented on AIRFLOW-121:
-

Commit 150568228b6f1321db29b92808945fb4b4cf932b in incubator-airflow's branch 
refs/heads/master from [~dud]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1505682 ]

[AIRFLOW-121] Documenting dag doc_md feature


> Documenting dag doc_md feature
> --
>
> Key: AIRFLOW-121
> URL: https://issues.apache.org/jira/browse/AIRFLOW-121
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: docs
>Reporter: dud
>Priority: Trivial
>
> Dear Airflow Maintainers,
> I added a note about DAG documentation.
> I'd be glad if my PR would be merged : 
> https://github.com/apache/incubator-airflow/pull/1493
> Regards
> dud



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-109) PrestoHook get_pandas_df executes a method that can raise outside of the try catch statement.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285327#comment-15285327
 ] 

ASF subversion and git services commented on AIRFLOW-109:
-

Commit 6f4696ba2ef18d74be8c18080b8ea7b9419608fb in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6f4696b ]

[AIRFLOW-109] Fix try catch handling in PrestoHook

This addresses the issue with executing the SQL statement outside of
the try block. In the case of a syntax error in the statement, the
underlying library raises a Databases error which was meant to be
handled (i.e., json parsed) by the catch.


> PrestoHook get_pandas_df executes a method that can raise outside of the try 
> catch statement.
> -
>
> Key: AIRFLOW-109
> URL: https://issues.apache.org/jira/browse/AIRFLOW-109
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Affects Versions: Airflow 1.8, Airflow 1.7.1, Airflow 1.6.2
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>  Labels: Presto
>
> This issue occurs when a malformed SQL statement is passed to the 
> get_pandas_df method of the presto hook. Pyhive raises a DatabaseError 
> outside of the try catch, leading in the wrong kind of error being raised.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-112) Change default view from graph view to tree view

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283218#comment-15283218
 ] 

ASF subversion and git services commented on AIRFLOW-112:
-

Commit 17bcf10fe55dd50fc42ae858d67bd26ba08988fe in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=17bcf10 ]

[AIRFLOW-112] no-op README change to close this jira's PR


> Change default view from graph view to tree view
> 
>
> Key: AIRFLOW-112
> URL: https://issues.apache.org/jira/browse/AIRFLOW-112
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Minor
>
> Most users use tree view instead of graph view to look at task instances (as 
> the history shown is useful), so the default view should be the tree view.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-86) Python3: UI crash without wrapping dict.items() in list

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289525#comment-15289525
 ] 

ASF subversion and git services commented on AIRFLOW-86:


Commit af43db5af0880bfabb2b044121134c8f050e3ae8 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=af43db5 ]

[AIRFLOW-86] Wrap dict.items() in list for Py3 compatibility

Author: jlowin 

Closes #1483 from jlowin/AIRFLOW-86.


> Python3: UI crash without wrapping dict.items() in list
> ---
>
> Key: AIRFLOW-86
> URL: https://issues.apache.org/jira/browse/AIRFLOW-86
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.0
>Reporter: Jeremiah Lowin
>  Labels: Python3
>
> Found while investigating AIRFLOW-62. Randomly causes a catastrophic failure 
> with Postgres backend when sqlalchemy tries to write the generator object to 
> the database.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-86) Python3: UI crash without wrapping dict.items() in list

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-86?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289524#comment-15289524
 ] 

ASF subversion and git services commented on AIRFLOW-86:


Commit af43db5af0880bfabb2b044121134c8f050e3ae8 in incubator-airflow's branch 
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=af43db5 ]

[AIRFLOW-86] Wrap dict.items() in list for Py3 compatibility

Author: jlowin 

Closes #1483 from jlowin/AIRFLOW-86.


> Python3: UI crash without wrapping dict.items() in list
> ---
>
> Key: AIRFLOW-86
> URL: https://issues.apache.org/jira/browse/AIRFLOW-86
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.0
>Reporter: Jeremiah Lowin
>  Labels: Python3
>
> Found while investigating AIRFLOW-62. Randomly causes a catastrophic failure 
> with Postgres backend when sqlalchemy tries to write the generator object to 
> the database.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-92) Tasks not being retried at all due to a 'obj not bound to a Session' exception

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278062#comment-15278062
 ] 

ASF subversion and git services commented on AIRFLOW-92:


Commit dddfd3b5bf2cabaac6eec123dfa3cb59e73a56f5 in incubator-airflow's branch 
refs/heads/master from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dddfd3b ]

AIRFLOW-92 Avoid unneeded upstream_failed session closes 
apache/incubator-airflow#1485


> Tasks not being retried at all due to a 'obj not bound to a Session' exception
> --
>
> Key: AIRFLOW-92
> URL: https://issues.apache.org/jira/browse/AIRFLOW-92
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.0
> Environment: EC2 t2.medium instance, 
> Docker `version 1.11.1, build 5604cbe`,
> Host is `Linux ip-172-31-44-140 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 
> 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux`,
> Docker containers are built upon the `python:3.5` image, 
> LocalExecutor is used with two scheduler containers running
>Reporter: Bence Nagy
>Priority: Critical
>
> I have some tasks that are stuck in {{up_for_retry}} state, below is an 
> extract from the database. (here it is in a [Google Drive 
> spreadsheet|https://docs.google.com/spreadsheets/d/14dtb3zYa583V1SaLcpOq6hDM4ThCeN7JhHjftRwKxbI/edit?usp=sharing]
>  with better formatting)
> {code}
> task_id   dag_id  execution_date  start_date  end_date
> durationstate   try_number  hostnameunixname
> job_id  poolqueue   priority_weight operatorqueued_dttm id
>   dag_id  state   job_typestart_date  end_date
> latest_heartbeatexecutor_class  hostnameunixnameid
>   dag_id  execution_date  state   run_id  external_triggerconf
> end_datestart_date
> task_adag_a1  2016-05-09 08:00:00.00  2016-05-09 
> 12:00:12.382775  2016-05-09 12:01:12.473914  60.091139   
> up_for_retry1   d5593c115c22root46266   default 4 
>   ExternalTaskSensor  46266   success LocalTaskJob
> 2016-05-09 12:00:08.195711  2016-05-09 12:01:13.261937  2016-05-09 
> 12:00:08.195732  LocalExecutor   d5593c115c22root17799   dag_a1  
> 2016-05-09 08:00:00.00  failed  scheduled__2016-05-09T08:00:00  false 
>   2016-05-09 12:00:04.406875
> task_adag_a2  2016-05-09 10:00:00.00  2016-05-09 
> 12:00:13.102094  2016-05-09 12:01:13.185960  60.083866   
> up_for_retry1   d5593c115c22root46270   default 4 
>   ExternalTaskSensor  46270   success LocalTaskJob
> 2016-05-09 12:00:08.896527  2016-05-09 12:01:13.960936  2016-05-09 
> 12:00:08.896550  LocalExecutor   d5593c115c22root17800   dag_a2  
> 2016-05-09 10:00:00.00  failed  scheduled__2016-05-09T10:00:00  false 
>   2016-05-09 12:00:04.531888
> task_bdag_b   2016-04-07 18:00:00.00  2016-05-09 
> 12:53:59.990395  2016-05-09 12:54:00.393259  0.402864
> up_for_retry1   0a8613c2b5d2root46366   default 1 
>   PostgresOperator46366   success LocalTaskJob
> 2016-05-09 12:53:58.881987  2016-05-09 12:54:03.891450  2016-05-09 
> 12:53:58.882006  LocalExecutor   0a8613c2b5d2root17836   dag_b   
> 2016-04-07 18:00:00.00  running scheduled__2016-04-07T18:00:00  false 
>   2016-05-09 12:51:59.713718
> task_cdag_b   2016-04-07 16:00:00.00  2016-05-09 
> 12:53:49.822634  2016-05-09 12:54:49.924291  60.101657   
> up_for_retry1   0a8613c2b5d2root46359   default 2 
>   ExternalTaskSensor  46359   success LocalTaskJob
> 2016-05-09 12:53:44.739355  2016-05-09 12:54:54.810579  2016-05-09 
> 12:53:44.739575  LocalExecutor   0a8613c2b5d2root17831   dag_b   
> 2016-04-07 16:00:00.00  running scheduled__2016-04-07T16:00:00  false 
>   2016-05-09 12:51:55.078050
> {code}
> I'm getting the following exception which seems to be halting the scheduler 
> just before it could queue the tasks for retrying:
> {code}
> [2016-05-10 09:42:33,562] {jobs.py:706} ERROR - Instance  0x7f48a6b87550> is not bound to a Session; attribute refresh operation cannot 
> proceed
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/site-packages/airflow/jobs.py", line 703, in 
> _do_dags
> self.process_dag(dag, tis_out)
>   File 

[jira] [Commented] (AIRFLOW-92) Tasks not being retried at all due to a 'obj not bound to a Session' exception

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-92?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278493#comment-15278493
 ] 

ASF subversion and git services commented on AIRFLOW-92:


Commit 4a5f4a0badfde97bd158f53513dab17450f8658b in incubator-airflow's branch 
refs/heads/airbnb_rb1.7.1_3 from [~bolke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4a5f4a0 ]

AIRFLOW-92 Avoid unneeded upstream_failed session closes 
apache/incubator-airflow#1485


> Tasks not being retried at all due to a 'obj not bound to a Session' exception
> --
>
> Key: AIRFLOW-92
> URL: https://issues.apache.org/jira/browse/AIRFLOW-92
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.0
> Environment: EC2 t2.medium instance, 
> Docker `version 1.11.1, build 5604cbe`,
> Host is `Linux ip-172-31-44-140 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 
> 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux`,
> Docker containers are built upon the `python:3.5` image, 
> LocalExecutor is used with two scheduler containers running
>Reporter: Bence Nagy
>Priority: Critical
> Fix For: Airflow 1.7.1
>
>
> I have some tasks that are stuck in {{up_for_retry}} state, below is an 
> extract from the database. (here it is in a [Google Drive 
> spreadsheet|https://docs.google.com/spreadsheets/d/14dtb3zYa583V1SaLcpOq6hDM4ThCeN7JhHjftRwKxbI/edit?usp=sharing]
>  with better formatting)
> {code}
> task_id   dag_id  execution_date  start_date  end_date
> durationstate   try_number  hostnameunixname
> job_id  poolqueue   priority_weight operatorqueued_dttm id
>   dag_id  state   job_typestart_date  end_date
> latest_heartbeatexecutor_class  hostnameunixnameid
>   dag_id  execution_date  state   run_id  external_triggerconf
> end_datestart_date
> task_adag_a1  2016-05-09 08:00:00.00  2016-05-09 
> 12:00:12.382775  2016-05-09 12:01:12.473914  60.091139   
> up_for_retry1   d5593c115c22root46266   default 4 
>   ExternalTaskSensor  46266   success LocalTaskJob
> 2016-05-09 12:00:08.195711  2016-05-09 12:01:13.261937  2016-05-09 
> 12:00:08.195732  LocalExecutor   d5593c115c22root17799   dag_a1  
> 2016-05-09 08:00:00.00  failed  scheduled__2016-05-09T08:00:00  false 
>   2016-05-09 12:00:04.406875
> task_adag_a2  2016-05-09 10:00:00.00  2016-05-09 
> 12:00:13.102094  2016-05-09 12:01:13.185960  60.083866   
> up_for_retry1   d5593c115c22root46270   default 4 
>   ExternalTaskSensor  46270   success LocalTaskJob
> 2016-05-09 12:00:08.896527  2016-05-09 12:01:13.960936  2016-05-09 
> 12:00:08.896550  LocalExecutor   d5593c115c22root17800   dag_a2  
> 2016-05-09 10:00:00.00  failed  scheduled__2016-05-09T10:00:00  false 
>   2016-05-09 12:00:04.531888
> task_bdag_b   2016-04-07 18:00:00.00  2016-05-09 
> 12:53:59.990395  2016-05-09 12:54:00.393259  0.402864
> up_for_retry1   0a8613c2b5d2root46366   default 1 
>   PostgresOperator46366   success LocalTaskJob
> 2016-05-09 12:53:58.881987  2016-05-09 12:54:03.891450  2016-05-09 
> 12:53:58.882006  LocalExecutor   0a8613c2b5d2root17836   dag_b   
> 2016-04-07 18:00:00.00  running scheduled__2016-04-07T18:00:00  false 
>   2016-05-09 12:51:59.713718
> task_cdag_b   2016-04-07 16:00:00.00  2016-05-09 
> 12:53:49.822634  2016-05-09 12:54:49.924291  60.101657   
> up_for_retry1   0a8613c2b5d2root46359   default 2 
>   ExternalTaskSensor  46359   success LocalTaskJob
> 2016-05-09 12:53:44.739355  2016-05-09 12:54:54.810579  2016-05-09 
> 12:53:44.739575  LocalExecutor   0a8613c2b5d2root17831   dag_b   
> 2016-04-07 16:00:00.00  running scheduled__2016-04-07T16:00:00  false 
>   2016-05-09 12:51:55.078050
> {code}
> I'm getting the following exception which seems to be halting the scheduler 
> just before it could queue the tasks for retrying:
> {code}
> [2016-05-10 09:42:33,562] {jobs.py:706} ERROR - Instance  0x7f48a6b87550> is not bound to a Session; attribute refresh operation cannot 
> proceed
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.5/site-packages/airflow/jobs.py", line 703, in 
> _do_dags
> self.process_dag(dag, tis_out)
>   

[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277355#comment-15277355
 ] 

ASF subversion and git services commented on AIRFLOW-52:


Commit ff3a8552501ac88821f3ce641471cd396ca93d6b in incubator-airflow's branch 
refs/heads/airbnb_rb1.7.1_3 from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ff3a855 ]

AIRFLOW-52 Warn about overwriting tasks in a DAG


> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
>  Issue Type: Task
>  Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>  Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added 
> more than once.
> 3. Airflow losing queued tasks (see linked issue)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a 
> given non-trivial DAG parses under X seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-355) "Dag only exists locally" message is never displayed

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394141#comment-15394141
 ] 

ASF subversion and git services commented on AIRFLOW-355:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


> "Dag only exists locally" message is never displayed
> 
>
> Key: AIRFLOW-355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-355
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> Because of the `and False` here 
> https://github.com/apache/incubator-airflow/blob/master/airflow/www/templates/airflow/dags.html#L72,
>  the message is never displayed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-354) info-sign glyphicon is used for edit dag button -

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394147#comment-15394147
 ] 

ASF subversion and git services commented on AIRFLOW-354:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


> info-sign glyphicon is used for edit dag button - 
> --
>
> Key: AIRFLOW-354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-354
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> The info-sign glyphicon is used for the edit dag button (leftmost column on 
> the front page). The edit glyphicon should be used instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-354) info-sign glyphicon is used for edit dag button -

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394148#comment-15394148
 ] 

ASF subversion and git services commented on AIRFLOW-354:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


> info-sign glyphicon is used for edit dag button - 
> --
>
> Key: AIRFLOW-354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-354
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> The info-sign glyphicon is used for the edit dag button (leftmost column on 
> the front page). The edit glyphicon should be used instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-356) is used on the front page

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394144#comment-15394144
 ] 

ASF subversion and git services commented on AIRFLOW-356:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


>  is used on the front page
> 
>
> Key: AIRFLOW-356
> URL: https://issues.apache.org/jira/browse/AIRFLOW-356
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> The  tag, which is non-standard and not on a standards track (per 
> https://developer.mozilla.org/en-US/docs/Web/HTML/Element/nobr), is used on 
> the front admin page. Alternatives like flexbox can be used instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-354) info-sign glyphicon is used for edit dag button -

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394142#comment-15394142
 ] 

ASF subversion and git services commented on AIRFLOW-354:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


> info-sign glyphicon is used for edit dag button - 
> --
>
> Key: AIRFLOW-354
> URL: https://issues.apache.org/jira/browse/AIRFLOW-354
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> The info-sign glyphicon is used for the edit dag button (leftmost column on 
> the front page). The edit glyphicon should be used instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-356) is used on the front page

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394143#comment-15394143
 ] 

ASF subversion and git services commented on AIRFLOW-356:
-

Commit a89d1015ceb6abccd53572bf38a44544d0182ca9 in incubator-airflow's branch 
refs/heads/master from [~xuanji]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a89d101 ]

[AIRFLOW-356][AIRFLOW-355][AIRFLOW-354] Replace nobr, enable DAG only exists 
locally message, change edit DAG icon

Addresses the following issues:
- 
[https://issues.apache.org/jira/browse/AIRFLOW-356](https://issues.apache.org/jira/browse/AIRFLOW-356)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-355](https://issues.apache.org/jira/browse/AIRFLOW-355)
- 
[https://issues.apache.org/jira/browse/AIRFLOW-354](https://issues.apache.org/jira/browse/AIRFLOW-354)

- Replace `` with `flexbox`
- "This DAG seems to be existing only locally" now shows up
- Change edit DAG icon from info to edit
- Rename `dttm` variable to `file_last_changed_on_disk`
- Rename `dags` variable to `webserver_dags`
- Adds a comment clarifying what `self.file_last_changed` is
- Clarifies what the `dag.last_expired` column represents
- Refactors some previously very nested logic in `views.py` and adds comments
- Properly indents `dags.html` and adds comments to it

- Edit DAG icon changed
- Home page now sort of responsive, no longer fixed width
- User will occasionally see "This DAG seems to be existing only locally" 
message

- Verify that edit dag button is now an edit icon and click on it
- Resized home page, check that last column does not wrap

![image](https://cloud.githubusercontent.com/assets/130362/17126889/2e7adb12-52b6-11e6-9a18-b31e424e4be8.png)

Clean up html, replace nobr with flexbox

Refactor HomeView

Rename variables and update comments

Closes #1678 from zodiac/xuanji_refactor


>  is used on the front page
> 
>
> Key: AIRFLOW-356
> URL: https://issues.apache.org/jira/browse/AIRFLOW-356
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Trivial
>
> The  tag, which is non-standard and not on a standards track (per 
> https://developer.mozilla.org/en-US/docs/Web/HTML/Element/nobr), is used on 
> the front admin page. Alternatives like flexbox can be used instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-335) Fix simple style errors/warnings

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378493#comment-15378493
 ] 

ASF subversion and git services commented on AIRFLOW-335:
-

Commit dc1bdf6222094e908b75056f9d6b0d14419e8777 in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=dc1bdf6 ]

[AIRFLOW-335] Fix simple style errors/warnings

Closes #1665 from skudriashev/airflow-335


> Fix simple style errors/warnings
> 
>
> Key: AIRFLOW-335
> URL: https://issues.apache.org/jira/browse/AIRFLOW-335
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: Airflow 1.7.1.2
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
>Priority: Trivial
> Fix For: Airflow 1.8
>
>
> Good idea to do clean-up style from time to time :)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-315) Fix all 'expected 2 blank lines, found ...' code style warning

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376816#comment-15376816
 ] 

ASF subversion and git services commented on AIRFLOW-315:
-

Commit 6c7c52c4475568560f394df0c51992937e678d0d in incubator-airflow's branch 
refs/heads/master from [~skudriashev]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6c7c52c ]

[AIRFLOW-315] Fix blank lines code style warnings

Closes #1653 from skudriashev/airflow-315


> Fix all 'expected 2 blank lines, found ...' code style warning
> --
>
> Key: AIRFLOW-315
> URL: https://issues.apache.org/jira/browse/AIRFLOW-315
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: Airflow 1.7.1.2
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
>Priority: Trivial
>
> Fix all 'expected 2 blank lines, found ...' code style warning to make code 
> style a bit better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-340) Unused Babel dependency

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388096#comment-15388096
 ] 

ASF subversion and git services commented on AIRFLOW-340:
-

Commit 00a591f7bff62cbb5fb5917960c60445d6d5abfb in incubator-airflow's branch 
refs/heads/master from [~kdeldycke]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=00a591f ]

[AIRFLOW-340] Remove unused dependency on Babel

Closes #1668 from kdeldycke/remove-unused-babel-deps


> Unused Babel dependency
> ---
>
> Key: AIRFLOW-340
> URL: https://issues.apache.org/jira/browse/AIRFLOW-340
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: release
>Affects Versions: Airflow 1.7.1.3
>Reporter: Kevin Deldycke
>Priority: Trivial
>  Labels: dependencies, packaging, patch
>
> I was just looking at the {{setup.py}} file and saw the dependency on 
> {{babel}}.
> The thing is, it doesn't look like Airflow is directly dependent on Babel 
> itself. See: 
> https://github.com/apache/incubator-airflow/search?utf8=%E2%9C%93=babel
> I propose to remove this dependency plain and simple.
> PR is available at: https://github.com/apache/incubator-airflow/pull/1668



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-349) Metric for zombies killed

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388103#comment-15388103
 ] 

ASF subversion and git services commented on AIRFLOW-349:
-

Commit b6e609824cb26bce1c9855f5d7e78a159ed03cc9 in incubator-airflow's branch 
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b6e6098 ]

[AIRFLOW-349] Add metric for number of zombies killed

Closes #1673 from aoen/ddavydov/metric_for_number_of_zombies_killed


> Metric for zombies killed
> -
>
> Key: AIRFLOW-349
> URL: https://issues.apache.org/jira/browse/AIRFLOW-349
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Minor
>
> There should be a metric exported every time a zombie (job that can't 
> heartbeat the scheduler) is killed, as if the number of zombies killed spikes 
> it will cause delayed tasks/spurious failure emails/multiple copies of a 
> taskinstance running at the same time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-331) correct the airflow online documentation on "Security / Web Authentication / LDAP " section

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396417#comment-15396417
 ] 

ASF subversion and git services commented on AIRFLOW-331:
-

Commit 7dbc3cd40e5c4f072897bcae46929ad61a2b4590 in incubator-airflow's branch 
refs/heads/master from Peter Pang
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7dbc3cd ]

[AIRFLOW-331] modify the LDAP authentication config lines in  'Security' sample 
codes

Closes #1674 from impangt/master


> correct the airflow online documentation on "Security / Web Authentication / 
> LDAP " section
> ---
>
> Key: AIRFLOW-331
> URL: https://issues.apache.org/jira/browse/AIRFLOW-331
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: docs
>Affects Versions: Airflow 1.7.1.3
>Reporter: peter pang
>Assignee: peter pang
>Priority: Minor
>  Labels: documentation
> Fix For: Airflow 1.8
>
>
> I'm trying to setup LDAP authentication for airflow 1.7.1.3 
> I added the simple [ldap] config example showing onling to my airflow.cfg 
> file, and the ldap login couldn't work. 
> After debug the auth_ldap.py I found "user_name_attr" and "search_scope"  are 
> not the setting value.  
> in the auth_ldap.py , configuration.get("ldap", "user_name_attr") return 
> value is the whole line strings " uid # in case of Active Directory you would 
> use" , include "#..." !!
> 
> [ldap]
> .
> user_name_attr = uid # in case of Active Directory you would use 
> 
> search_scope = LEVEL # Set this to SUBTREE if using Active Directory, and not 
> specifying an Organizational Unit
> =
> so the comment lines shouldn't be there, they must be in a new line with no 
> space or tab char before the '#'
> 
> [ldap]
> 
> "#" in case of Active Directory you would use 
> user_name_attr = uid
> ...
> "#" Set this to SUBTREE if using Active Directory, and not specifying an 
> Organizational Unit
> search_scope = LEVEL
> =
> then ldap could work. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-381) Manual UI Dag Run creation: require dag_id field

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398568#comment-15398568
 ] 

ASF subversion and git services commented on AIRFLOW-381:
-

Commit 35c2215b9f0733853592ccfc88119e3102af610a in incubator-airflow's branch 
refs/heads/master from [~normster]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=35c2215 ]

[AIRFLOW-381] Manual UI Dag Run creation: require dag_id field


> Manual UI Dag Run creation: require dag_id field
> 
>
> Key: AIRFLOW-381
> URL: https://issues.apache.org/jira/browse/AIRFLOW-381
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: Airflow 1.7.1.3
>Reporter: Norman Mu
>Assignee: Norman Mu
>Priority: Minor
>
> Currently the dag_id field is not required when manually creating a dag run 
> in the UI, which if left blank will error in the scheduler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-383) Fix example_qubole_operator DAG

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402159#comment-15402159
 ] 

ASF subversion and git services commented on AIRFLOW-383:
-

Commit eb989dcb54387c218683de9dff0a12950dce147b in incubator-airflow's branch 
refs/heads/master from [~yogeshg]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=eb989dc ]

[AIRFLOW-383] Cleanup example qubole operator dag

Closes #1698 from yogesh2021/AIRFLOW-383


> Fix example_qubole_operator DAG
> ---
>
> Key: AIRFLOW-383
> URL: https://issues.apache.org/jira/browse/AIRFLOW-383
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Yogesh Garg
>Assignee: Yogesh Garg
>Priority: Minor
> Fix For: Airflow 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-362) Chunked transfers of files larger than 5GB truncate the transferred files

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393817#comment-15393817
 ] 

ASF subversion and git services commented on AIRFLOW-362:
-

Commit 2c931abb36826f96caefcc2663da8262816a5b1d in incubator-airflow's branch 
refs/heads/master from [~clearclaw]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2c931ab ]

[AIRFLOW-362] Import __future__ division

Fix integer division fencepost error on S3 chunked transfers

Closes #1683 from clearclaw/master


> Chunked transfers of files larger than 5GB truncate the transferred files
> -
>
> Key: AIRFLOW-362
> URL: https://issues.apache.org/jira/browse/AIRFLOW-362
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: J C Lawrence
>Assignee: J C Lawrence
>
> S3_hook.py:load_file(346) has a fencepost error under Py2 due to integer 
> division versus floating point division (in Py3).  The result is that the 
> last block of large files (>5GB) won't be transferred.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-160) Parse DAG files through child processes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401332#comment-15401332
 ] 

ASF subversion and git services commented on AIRFLOW-160:
-

Commit fdb7e949140b735b8554ae5b22ad752e86f6ebaf in incubator-airflow's branch 
refs/heads/master from [~pauly]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fdb7e94 ]

[AIRFLOW-160] Parse DAG files through child processes

Instead of parsing the DAG definition files in the same process as the
scheduler, this change parses the files in a child process. This helps
to isolate the scheduler from bad user code.

Closes #1636 from plypaul/plypaul_schedule_by_file_rebase_master


> Parse DAG files through child processes
> ---
>
> Key: AIRFLOW-160
> URL: https://issues.apache.org/jira/browse/AIRFLOW-160
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Reporter: Paul Yang
>Assignee: Paul Yang
>
> Currently, the Airflow scheduler parses all user DAG files in the same 
> process as the scheduler itself. We've seen issues in production where bad 
> DAG files cause scheduler to fail. A simple example is if the user script 
> calls `sys.exit(1)`, the scheduler will exit as well. We've also seen an 
> unusual case where modules loaded by the user DAG affect operation of the 
> scheduler. For better uptime, the scheduler should be resistant to these 
> problematic user DAGs.
> The proposed solution is to parse and schedule user DAGs through child 
> processes. This way, the main scheduler process is more isolated from bad 
> DAGs. There's a side benefit as well - since parsing is distributed among 
> multiple processes, it's possible to parse the DAG files more frequently, 
> reducing the latency between when a DAG is modified and when the changes are 
> picked up.
> Another issue right now is that all DAGs must be scheduled before any tasks 
> are sent to the executor. This means that the frequency of task scheduling is 
> limited by the slowest DAG to schedule. The changes needed for scheduling 
> DAGs through child processes will also make it easy to decouple this process 
> and allow tasks to be scheduled and sent to the executor in a more 
> independent fashion. This way, overall scheduling won't be held back by a 
> slow DAG.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)