[jira] [Updated] (AIRFLOW-285) Update remaining imports to use 2.0 style

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-285:

External issue URL: https://github.com/apache/incubator-airflow/pull/1586

> Update remaining imports to use 2.0 style
> -
>
> Key: AIRFLOW-285
> URL: https://issues.apache.org/jira/browse/AIRFLOW-285
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: Airflow 1.7.1.3
>Reporter: Chris Riccomini
>Assignee: Chris Riccomini
> Fix For: Airflow 1.8
>
>
> As part of AIRFLOW-200, I noticed that not all imports were updated in 
> AIRFLOW-31. Some are still using old style imports (e.g. {{from 
> airflow.operators import BashOperator}}). This causes a problem with my 
> AIRFLOW-200 patch, as it assumes if new style imports are used, they'll be 
> used for all hooks/operators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-285) Update remaining imports to use 2.0 style

[Chris Riccomini (JIRA)]

Chris Riccomini created AIRFLOW-285:
---

 Summary: Update remaining imports to use 2.0 style
 Key: AIRFLOW-285
 URL: https://issues.apache.org/jira/browse/AIRFLOW-285
 Project: Apache Airflow
  Issue Type: Bug
  Components: core
Affects Versions: Airflow 1.7.1.3
Reporter: Chris Riccomini
Assignee: Chris Riccomini
 Fix For: Airflow 1.8


As part of AIRFLOW-200, I noticed that not all imports were updated in 
AIRFLOW-31. Some are still using old style imports (e.g. {{from 
airflow.operators import BashOperator}}). This causes a problem with my 
AIRFLOW-200 patch, as it assumes if new style imports are used, they'll be used 
for all hooks/operators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-247) EMR Hook, Operators, Sensor

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352253#comment-15352253
 ] 

Chris Riccomini commented on AIRFLOW-247:
-

[~gjreda], want to have a look at the PR?

> EMR Hook, Operators, Sensor
> ---
>
> Key: AIRFLOW-247
> URL: https://issues.apache.org/jira/browse/AIRFLOW-247
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Rob Froetscher
>Assignee: Rob Froetscher
>Priority: Minor
>
> Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be 
> nice to have an EMR hook and operators.
> Hook to generally interact with EMR.
> Operators to:
> * setup and start a job flow
> * add steps to an existing jobflow 
> A sensor to:
> * monitor completion and status of EMR jobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-277) Multiple deletions does not work in Task Instances view if using SQLite backend

[Kengo Seki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kengo Seki updated AIRFLOW-277:
---
  Assignee: Kengo Seki
External issue URL: https://github.com/apache/incubator-airflow/pull/1631

> Multiple deletions does not work in Task Instances view if using SQLite 
> backend
> ---
>
> Key: AIRFLOW-277
> URL: https://issues.apache.org/jira/browse/AIRFLOW-277
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.7.1.3
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>
> AIRFLOW-252 fixed the trash button on Task Instances view, but "With 
> selected" > "Delete" menu still seems not working on SQLite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-284) HiveHook Cursor Scope Persistency

[Sherwain Williamson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sherwain Williamson updated AIRFLOW-284:

Description: 
The function `get_results` implemented in the `HiveServer2Hook` does not 
execute multiple commands passed to it in a list, in the singular cursor scope. 
This has caused SQL statements that depend on the execution of add `jar` and 
`set` commands to fail as they are being executed in different cursor scopes 
which are not persistent.
The code has been updated to have the cursor object persistent.

  was:
The function `get_results` implemented in the `HiveServer2Hook` does not 
execute multiple commands passed to it, in a list, in the singular cursor 
scope. This has caused SQL statements that depends on the execution of `add 
jar' and `set` command to fail as they are being executed in the different 
cursor scope and they are not persistent.
The code has been updated to have the cursor object persistent.


> HiveHook Cursor Scope Persistency
> -
>
> Key: AIRFLOW-284
> URL: https://issues.apache.org/jira/browse/AIRFLOW-284
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hive_hooks, hooks
>Affects Versions: Airflow 2.0
> Environment: Apache
>Reporter: Sherwain Williamson
>Assignee: Sherwain Williamson
> Fix For: Airflow 2.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The function `get_results` implemented in the `HiveServer2Hook` does not 
> execute multiple commands passed to it in a list, in the singular cursor 
> scope. This has caused SQL statements that depend on the execution of add 
> `jar` and `set` commands to fail as they are being executed in different 
> cursor scopes which are not persistent.
> The code has been updated to have the cursor object persistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-247) EMR Hook, Operators, Sensor

[Rob Froetscher (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352017#comment-15352017
 ] 

Rob Froetscher commented on AIRFLOW-247:


https://github.com/apache/incubator-airflow/pull/1630

> EMR Hook, Operators, Sensor
> ---
>
> Key: AIRFLOW-247
> URL: https://issues.apache.org/jira/browse/AIRFLOW-247
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Rob Froetscher
>Assignee: Rob Froetscher
>Priority: Minor
>
> Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be 
> nice to have an EMR hook and operators.
> Hook to generally interact with EMR.
> Operators to:
> * setup and start a job flow
> * add steps to an existing jobflow 
> A sensor to:
> * monitor completion and status of EMR jobs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-284) HiveHook Cursor Scope Persistency

[Sherwain Williamson (JIRA)]

Sherwain Williamson created AIRFLOW-284:
---

 Summary: HiveHook Cursor Scope Persistency
 Key: AIRFLOW-284
 URL: https://issues.apache.org/jira/browse/AIRFLOW-284
 Project: Apache Airflow
  Issue Type: Bug
  Components: hive_hooks, hooks
Affects Versions: Airflow 2.0
 Environment: Apache
Reporter: Sherwain Williamson
Assignee: Sherwain Williamson
 Fix For: Airflow 2.0


The function `get_results` implemented in the `HiveServer2Hook` does not 
execute multiple commands passed to it, in a list, in the singular cursor 
scope. This has caused SQL statements that depends on the execution of `add 
jar' and `set` command to fail as they are being executed in the different 
cursor scope and they are not persistent.
The code has been updated to have the cursor object persistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-283) Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-283.
---
   Resolution: Fixed
 Assignee: Ilya Rakoshes
Fix Version/s: Airflow 1.8

+1 Merged

> Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator
> --
>
> Key: AIRFLOW-283
> URL: https://issues.apache.org/jira/browse/AIRFLOW-283
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Ilya Rakoshes
>Assignee: Ilya Rakoshes
>Priority: Minor
> Fix For: Airflow 1.8
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-283) Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-283:

External issue URL: https://github.com/apache/incubator-airflow/pull/1627

> Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator
> --
>
> Key: AIRFLOW-283
> URL: https://issues.apache.org/jira/browse/AIRFLOW-283
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Ilya Rakoshes
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-281) mssql_hook does not pass port to pymssql's connection method.

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-281:

External issue URL: https://github.com/apache/incubator-airflow/pull/1626

> mssql_hook does not pass port to pymssql's connection method. 
> --
>
> Key: AIRFLOW-281
> URL: https://issues.apache.org/jira/browse/AIRFLOW-281
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: Luke Maycock
>Assignee: Luke Maycock
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> The mssql_hook does not pass the port through to pymssql's connection method. 
> Therefore, the port entered in the UI in 'Admin > Connections' is ignored for 
> mssql connections. 
> Proposed fix: 
> Used named arguments and include the 'port' argument. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-281) mssql_hook does not pass port to pymssql's connection method.

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351401#comment-15351401
 ] 

Chris Riccomini commented on AIRFLOW-281:
-

[~lukem], want to send a PR for this?

> mssql_hook does not pass port to pymssql's connection method. 
> --
>
> Key: AIRFLOW-281
> URL: https://issues.apache.org/jira/browse/AIRFLOW-281
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: Luke Maycock
>Assignee: Luke Maycock
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> The mssql_hook does not pass the port through to pymssql's connection method. 
> Therefore, the port entered in the UI in 'Admin > Connections' is ignored for 
> mssql connections. 
> Proposed fix: 
> Used named arguments and include the 'port' argument. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-276) List of dags does not refresh in UI for a while

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-276:

External issue URL: https://github.com/apache/incubator-airflow/pull/1621

> List of dags does not refresh in UI for a while
> ---
>
> Key: AIRFLOW-276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-276
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Minor
>
> After creating a new dag (eg by adding a file to `~/airflow/dags`), the web 
> UI does not show the new for a while. It only shows it when either
> 1. gunicorn decides to restart the worker process, or
> 2. a scheduler picks up the new dag, adds it to the airflow db, and the web 
> UI notices it in the db



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (AIRFLOW-276) List of dags does not refresh in UI for a while

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-276:

Comment: was deleted

(was: I believe that this is known behavior. The scheduler refreshes the DAG. 
You can click the "refresh" button in the UI to trigger a more immediate 
refresh.)

> List of dags does not refresh in UI for a while
> ---
>
> Key: AIRFLOW-276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-276
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Minor
>
> After creating a new dag (eg by adding a file to `~/airflow/dags`), the web 
> UI does not show the new for a while. It only shows it when either
> 1. gunicorn decides to restart the worker process, or
> 2. a scheduler picks up the new dag, adds it to the airflow db, and the web 
> UI notices it in the db



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-279) Dagrun timeouts not applied after max number of active runs is reached

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-279:

External issue URL: https://github.com/apache/incubator-airflow/pull/1623

> Dagrun timeouts not applied after max number of active runs is reached
> --
>
> Key: AIRFLOW-279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-279
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.1.3
>Reporter: Betsy Cannon
>Assignee: Betsy Cannon
>Priority: Minor
>
> `schedule_dag` checks against the max number of active runs and exits early 
> before cleaning up dagruns that have timed out. This means if you reach the 
> max number of runs, the timeouts are never applied and new dagruns are not 
> started. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-279) Dagrun timeouts not applied after max number of active runs is reached

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-279.
---
Resolution: Duplicate

Dupe according to the PR.

> Dagrun timeouts not applied after max number of active runs is reached
> --
>
> Key: AIRFLOW-279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-279
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.1.3
>Reporter: Betsy Cannon
>Assignee: Betsy Cannon
>Priority: Minor
>
> `schedule_dag` checks against the max number of active runs and exits early 
> before cleaning up dagruns that have timed out. This means if you reach the 
> max number of runs, the timeouts are never applied and new dagruns are not 
> started. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-280) clean up the tmp druid table no matter if an ingestion job succeeds or not

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-280.
---
Resolution: Fixed

> clean up the tmp druid table no matter if an ingestion job succeeds or not
> --
>
> Key: AIRFLOW-280
> URL: https://issues.apache.org/jira/browse/AIRFLOW-280
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: Airflow 1.8
>Reporter: Hongbo Zeng
>Assignee: Hongbo Zeng
> Fix For: Airflow 1.8
>
>
> The tmp table for Druid data will stay if a batch ingestion fails, we need to 
> clean up the hive table no matter an ingestion job succeeds or not



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-279) Dagrun timeouts not applied after max number of active runs is reached

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-279:

Component/s: scheduler

> Dagrun timeouts not applied after max number of active runs is reached
> --
>
> Key: AIRFLOW-279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-279
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Betsy Cannon
>Assignee: Betsy Cannon
>Priority: Minor
>
> `schedule_dag` checks against the max number of active runs and exits early 
> before cleaning up dagruns that have timed out. This means if you reach the 
> max number of runs, the timeouts are never applied and new dagruns are not 
> started. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-279) Dagrun timeouts not applied after max number of active runs is reached

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-279:

Affects Version/s: 1.7.1.3

> Dagrun timeouts not applied after max number of active runs is reached
> --
>
> Key: AIRFLOW-279
> URL: https://issues.apache.org/jira/browse/AIRFLOW-279
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Affects Versions: Airflow 1.7.1.3
>Reporter: Betsy Cannon
>Assignee: Betsy Cannon
>Priority: Minor
>
> `schedule_dag` checks against the max number of active runs and exits early 
> before cleaning up dagruns that have timed out. This means if you reach the 
> max number of runs, the timeouts are never applied and new dagruns are not 
> started. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-276) List of dags does not refresh in UI for a while

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351368#comment-15351368
 ] 

Chris Riccomini commented on AIRFLOW-276:
-

I believe that this is known behavior. The scheduler refreshes the DAG. You can 
click the "refresh" button in the UI to trigger a more immediate refresh.

> List of dags does not refresh in UI for a while
> ---
>
> Key: AIRFLOW-276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-276
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Minor
>
> After creating a new dag (eg by adding a file to `~/airflow/dags`), the web 
> UI does not show the new for a while. It only shows it when either
> 1. gunicorn decides to restart the worker process, or
> 2. a scheduler picks up the new dag, adds it to the airflow db, and the web 
> UI notices it in the db



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-276) List of dags does not refresh in UI for a while

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-276.
---
Resolution: Won't Fix

> List of dags does not refresh in UI for a while
> ---
>
> Key: AIRFLOW-276
> URL: https://issues.apache.org/jira/browse/AIRFLOW-276
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Li Xuanji
>Assignee: Li Xuanji
>Priority: Minor
>
> After creating a new dag (eg by adding a file to `~/airflow/dags`), the web 
> UI does not show the new for a while. It only shows it when either
> 1. gunicorn decides to restart the worker process, or
> 2. a scheduler picks up the new dag, adds it to the airflow db, and the web 
> UI notices it in the db



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AIRFLOW-272) Create a DagRun sensor

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-272:

External issue URL: https://github.com/apache/incubator-airflow/pull/1617

> Create a DagRun sensor
> --
>
> Key: AIRFLOW-272
> URL: https://issues.apache.org/jira/browse/AIRFLOW-272
> Project: Apache Airflow
>  Issue Type: New Feature
>Reporter: Narek Amirbekian
>Assignee: Narek Amirbekian
>Priority: Minor
>
> There is no way to tell a task or sub-dag to not run until a certain DagRun 
> is complete. We could implemnt a DagRunStateSensor similar to the 
> ExternalTaskSensor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-267) PythonOperators return value seems to be ignored

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-267.
---
Resolution: Not A Problem

> PythonOperators return value seems to be ignored
> 
>
> Key: AIRFLOW-267
> URL: https://issues.apache.org/jira/browse/AIRFLOW-267
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: Airflow 1.7.0
>Reporter: Eric Johnson
>Priority: Minor
>
> This is a simple dag that I don't think should run to completion as there as 
> a PythonOperator that returns False which should halt the dependency. Right?
> morning depends on check_false which depends on check_true.
> check_true's PythonOperator returns true. So that's fine. But check_false is 
> a PythonOperator that returns False. Shouldn't that halt the execution?
> {code}
> from builtins import range
> from airflow.operators import BashOperator, DummyOperator, TimeSensor, 
> PythonOperator
> from airflow.models import DAG
> from datetime import datetime, timedelta, time
> one_day_ago = datetime.combine(datetime.today() - timedelta(1), 
> datetime.min.time())
> args = {
> 'owner': 'ejohnson',
> 'start_date' : one_day_ago,
> 'email' : "ejohn...@example.com",
> 'email_on_failure' : True
> }
> # This is the master container for the mydag
> mydag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval=None
> )
> def check_func_true(ds, **kwargs):
> return True
> def check_func_false(ds, **kwargs):
> return False
> check_false = PythonOperator(
> task_id='check_false',
> provide_context=True,
> python_callable=check_func_false,
> email="ejohn...@example.com",
> email_on_retry=True,
> email_on_failure=True,
> retries=5,
> dag=mydag)
> check_true = PythonOperator(
> task_id='check_true',
> provide_context=True,
> python_callable=check_func_true,
> email="ejohn...@example.com",
> email_on_retry=True,
> email_on_failure=True,
> retries=5,
> dag=mydag)
> morning = BashOperator(task_id='morning',bash_command="echo 
> morning",dag=mydag)
> morning.set_upstream(check_false)
> check_false.set_upstream(check_true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-267) PythonOperators return value seems to be ignored

[Chris Riccomini (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351290#comment-15351290
 ] 

Chris Riccomini commented on AIRFLOW-267:
-

Returning false does not halt the execution of a DAG. Returned values are 
stored as XCom fields. From [XCom 
docs|http://airflow.incubator.apache.org/concepts.html#xcoms]:

{quote}
In addition, if a task returns a value (either from its Operator’s execute() 
method, or from a PythonOperator’s python_callable function), then an XCom 
containing that value is automatically pushed.
{quote}

You should have a look at the BranchOperator to circumvent logic that you don't 
want to run.

> PythonOperators return value seems to be ignored
> 
>
> Key: AIRFLOW-267
> URL: https://issues.apache.org/jira/browse/AIRFLOW-267
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: operators
>Affects Versions: Airflow 1.7.0
>Reporter: Eric Johnson
>Priority: Minor
>
> This is a simple dag that I don't think should run to completion as there as 
> a PythonOperator that returns False which should halt the dependency. Right?
> morning depends on check_false which depends on check_true.
> check_true's PythonOperator returns true. So that's fine. But check_false is 
> a PythonOperator that returns False. Shouldn't that halt the execution?
> {code}
> from builtins import range
> from airflow.operators import BashOperator, DummyOperator, TimeSensor, 
> PythonOperator
> from airflow.models import DAG
> from datetime import datetime, timedelta, time
> one_day_ago = datetime.combine(datetime.today() - timedelta(1), 
> datetime.min.time())
> args = {
> 'owner': 'ejohnson',
> 'start_date' : one_day_ago,
> 'email' : "ejohn...@example.com",
> 'email_on_failure' : True
> }
> # This is the master container for the mydag
> mydag = DAG(
> dag_id='mydag',
> default_args=args,
> schedule_interval=None
> )
> def check_func_true(ds, **kwargs):
> return True
> def check_func_false(ds, **kwargs):
> return False
> check_false = PythonOperator(
> task_id='check_false',
> provide_context=True,
> python_callable=check_func_false,
> email="ejohn...@example.com",
> email_on_retry=True,
> email_on_failure=True,
> retries=5,
> dag=mydag)
> check_true = PythonOperator(
> task_id='check_true',
> provide_context=True,
> python_callable=check_func_true,
> email="ejohn...@example.com",
> email_on_retry=True,
> email_on_failure=True,
> retries=5,
> dag=mydag)
> morning = BashOperator(task_id='morning',bash_command="echo 
> morning",dag=mydag)
> morning.set_upstream(check_false)
> check_false.set_upstream(check_true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-268) SSHExecuteOperator doesn't work on OSX

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini closed AIRFLOW-268.
---
Resolution: Duplicate

Dupe of AIRFLOW-130

> SSHExecuteOperator doesn't work on OSX
> --
>
> Key: AIRFLOW-268
> URL: https://issues.apache.org/jira/browse/AIRFLOW-268
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Vineet Goel
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> The SSHExecuteOperator runs a mktemp command using the --tempdir flag which 
> is compatible with Linux systems but doesn't work on mac OSX. A fix would be 
> to use a platform-independent version of the command as shown here:
> http://stackoverflow.com/questions/31396985/why-is-mktemp-on-os-x-broken-with-a-command-that-worked-on-linux



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (AIRFLOW-278) Support utf-8 ecoding for sql query strings

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin closed AIRFLOW-278.
--
Resolution: Fixed

Closed in https://github.com/apache/incubator-airflow/pull/1622

> Support utf-8 ecoding for sql query strings
> ---
>
> Key: AIRFLOW-278
> URL: https://issues.apache.org/jira/browse/AIRFLOW-278
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Nikolay Bilev
>Assignee: Nikolay Bilev
>
> 'utf-8' encoding for python 2 users who has unicode strings inside of sql 
> queries. Default 'ancii' codec doesn't handle them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-282) PR tool crashes when fix_version format is not A.B.C

[Jeremiah Lowin (JIRA)]

Jeremiah Lowin created AIRFLOW-282:
--

 Summary: PR tool crashes when fix_version format is not A.B.C
 Key: AIRFLOW-282
 URL: https://issues.apache.org/jira/browse/AIRFLOW-282
 Project: Apache Airflow
  Issue Type: Bug
  Components: PR tool
Affects Versions: 1.7.1.3
Reporter: Jeremiah Lowin
Assignee: Jeremiah Lowin


The PR tool expects a version number with the format major.minor.patch, but 
Airflow versions range from major.minor (2.0) to major.minor.patch.point 
(1.7.1.3). Until (and if) we standardize, this logic should be removed since it 
crashes the tool and doesn't add significant value anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AIRFLOW-278) Support utf-8 ecoding for sql query strings

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351224#comment-15351224
 ] 

ASF subversion and git services commented on AIRFLOW-278:
-

Commit 5db7ec70f8b45bc4872ae64901d2829efa0e4081 in incubator-airflow's branch 
refs/heads/master from [~bilevn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5db7ec7 ]

[AIRFLOW-278] Support utf-8 ecoding for SQL

Support utf-8 encoding for SQL queries, needed for Python 2
users who have unicode strings inside the queries

Closes #1622 from biln/master


> Support utf-8 ecoding for sql query strings
> ---
>
> Key: AIRFLOW-278
> URL: https://issues.apache.org/jira/browse/AIRFLOW-278
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Reporter: Nikolay Bilev
>Assignee: Nikolay Bilev
>
> 'utf-8' encoding for python 2 users who has unicode strings inside of sql 
> queries. Default 'ancii' codec doesn't handle them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-airflow git commit: [AIRFLOW-278] Support utf-8 ecoding for SQL

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 3b84bcb3e -> 5db7ec70f


[AIRFLOW-278] Support utf-8 ecoding for SQL

Support utf-8 encoding for SQL queries, needed for Python 2
users who have unicode strings inside the queries

Closes #1622 from biln/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/5db7ec70
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/5db7ec70
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/5db7ec70

Branch: refs/heads/master
Commit: 5db7ec70f8b45bc4872ae64901d2829efa0e4081
Parents: 3b84bcb
Author: Nikolay Bilev 
Authored: Mon Jun 27 11:20:36 2016 -0400
Committer: jlowin 
Committed: Mon Jun 27 11:20:41 2016 -0400

--
 airflow/hooks/dbapi_hook.py | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/5db7ec70/airflow/hooks/dbapi_hook.py
--
diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index f6e86b6..55f2d95 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -17,6 +17,7 @@ from past.builtins import basestring
 from datetime import datetime
 import numpy
 import logging
+import sys
 
 from airflow.hooks.base_hook import BaseHook
 from airflow.exceptions import AirflowException
@@ -66,6 +67,8 @@ class DbApiHook(BaseHook):
 :param parameters: The parameters to render the SQL query with.
 :type parameters: mapping or iterable
 '''
+if sys.version_info[0] < 3:
+sql = sql.encode('utf-8')
 import pandas.io.sql as psql
 conn = self.get_conn()
 df = psql.read_sql(sql, con=conn, params=parameters)
@@ -82,6 +85,8 @@ class DbApiHook(BaseHook):
 :param parameters: The parameters to render the SQL query with.
 :type parameters: mapping or iterable
 '''
+if sys.version_info[0] < 3:
+sql = sql.encode('utf-8')
 conn = self.get_conn()
 cur = self.get_cursor()
 if parameters is not None:
@@ -103,6 +108,8 @@ class DbApiHook(BaseHook):
 :param parameters: The parameters to render the SQL query with.
 :type parameters: mapping or iterable
 '''
+if sys.version_info[0] < 3:
+sql = sql.encode('utf-8')
 conn = self.get_conn()
 cur = conn.cursor()
 if parameters is not None:
@@ -138,6 +145,8 @@ class DbApiHook(BaseHook):
 
 cur = conn.cursor()
 for s in sql:
+if sys.version_info[0] < 3:
+s = s.encode('utf-8')
 logging.info(s)
 if parameters is not None:
 cur.execute(s, parameters)

[jira] [Work started] (AIRFLOW-281) mssql_hook does not pass port to pymssql's connection method.

[Luke Maycock (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-281 started by Luke Maycock.

> mssql_hook does not pass port to pymssql's connection method. 
> --
>
> Key: AIRFLOW-281
> URL: https://issues.apache.org/jira/browse/AIRFLOW-281
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hooks
>Reporter: Luke Maycock
>Assignee: Luke Maycock
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> The mssql_hook does not pass the port through to pymssql's connection method. 
> Therefore, the port entered in the UI in 'Admin > Connections' is ignored for 
> mssql connections. 
> Proposed fix: 
> Used named arguments and include the 'port' argument. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (AIRFLOW-281) mssql_hook does not pass port to pymssql's connection method.

[Luke Maycock (JIRA)]

Luke Maycock created AIRFLOW-281:


 Summary: mssql_hook does not pass port to pymssql's connection 
method. 
 Key: AIRFLOW-281
 URL: https://issues.apache.org/jira/browse/AIRFLOW-281
 Project: Apache Airflow
  Issue Type: Bug
  Components: hooks
Reporter: Luke Maycock
Assignee: Luke Maycock


The mssql_hook does not pass the port through to pymssql's connection method. 
Therefore, the port entered in the UI in 'Admin > Connections' is ignored for 
mssql connections. 

Proposed fix: 
Used named arguments and include the 'port' argument. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)