[jira] [Comment Edited] (AIRFLOW-923) airflow webserver -D flag doesn't daemonize anymore

2018-02-11 Thread Yuliya Volkova (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360386#comment-16360386
 ] 

Yuliya Volkova edited comment on AIRFLOW-923 at 2/12/18 7:34 AM:
-

checked in version 1.9.0

daemonization, seems work correct : airflow webserver -D 

screens attached

 


was (Author: xnuinside):
!Screen Shot 2018-02-12 at 10.32.23 AM.png!!Screen Shot 2018-02-12 at 10.32.33 
AM.png!

> airflow webserver -D flag doesn't daemonize anymore
> ---
>
> Key: AIRFLOW-923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-923
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: joyce chan
>Priority: Trivial
> Fix For: Airflow 1.8
>
> Attachments: Screen Shot 2018-02-12 at 10.32.23 AM.png, Screen Shot 
> 2018-02-12 at 10.32.33 AM.png
>
>
> Airflow 1.8 rc4
> airflow webserver -D flag doesn't daemonize anymore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-923) airflow webserver -D flag doesn't daemonize anymore

2018-02-11 Thread Yuliya Volkova (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360384#comment-16360384
 ] 

Yuliya Volkova commented on AIRFLOW-923:


[~joyceschan], please check version 1.9.0

> airflow webserver -D flag doesn't daemonize anymore
> ---
>
> Key: AIRFLOW-923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-923
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: joyce chan
>Priority: Trivial
> Fix For: Airflow 1.8
>
> Attachments: Screen Shot 2018-02-12 at 10.32.23 AM.png, Screen Shot 
> 2018-02-12 at 10.32.33 AM.png
>
>
> Airflow 1.8 rc4
> airflow webserver -D flag doesn't daemonize anymore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-923) airflow webserver -D flag doesn't daemonize anymore

2018-02-11 Thread Yuliya Volkova (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuliya Volkova updated AIRFLOW-923:
---
Attachment: Screen Shot 2018-02-12 at 10.32.33 AM.png
Screen Shot 2018-02-12 at 10.32.23 AM.png

> airflow webserver -D flag doesn't daemonize anymore
> ---
>
> Key: AIRFLOW-923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-923
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: joyce chan
>Priority: Trivial
> Fix For: Airflow 1.8
>
> Attachments: Screen Shot 2018-02-12 at 10.32.23 AM.png, Screen Shot 
> 2018-02-12 at 10.32.33 AM.png
>
>
> Airflow 1.8 rc4
> airflow webserver -D flag doesn't daemonize anymore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-923) airflow webserver -D flag doesn't daemonize anymore

2018-02-11 Thread Yuliya Volkova (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360386#comment-16360386
 ] 

Yuliya Volkova commented on AIRFLOW-923:


!Screen Shot 2018-02-12 at 10.32.23 AM.png!!Screen Shot 2018-02-12 at 10.32.33 
AM.png!

> airflow webserver -D flag doesn't daemonize anymore
> ---
>
> Key: AIRFLOW-923
> URL: https://issues.apache.org/jira/browse/AIRFLOW-923
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: joyce chan
>Priority: Trivial
> Fix For: Airflow 1.8
>
> Attachments: Screen Shot 2018-02-12 at 10.32.23 AM.png, Screen Shot 
> 2018-02-12 at 10.32.33 AM.png
>
>
> Airflow 1.8 rc4
> airflow webserver -D flag doesn't daemonize anymore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360136#comment-16360136
 ] 

Kaxil Naik commented on AIRFLOW-2100:
-

I have updated the Pythonhosted links with the docs at 
http://airflow.incubator.apache.org/ . This has been documented at 
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
 by Chris Riccomini.

I have also added the Readthedocs badge for the latest version. Have created a 
[PR|https://github.com/apache/incubator-airflow/pull/3033].

Regards,
Kaxil

> Links to the official docs are all broken
> -
>
> Key: AIRFLOW-2100
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs, Documentation
>Affects Versions: 1.9.0, 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.9.0, 2.0.0
>
>
> Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve 
> it:
> # Change the all the links to readthedocs 
> ([latest|http://airflow.readthedocs.io/en/latest/] / 
> [stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
> # Can someone who has access to PyPI create the docs?
> # Replace link to http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360136#comment-16360136
 ] 

Kaxil Naik edited comment on AIRFLOW-2100 at 2/11/18 9:30 PM:
--

I have updated the Pythonhosted links with the docs at 
http://airflow.incubator.apache.org/ . This has been documented at 
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
 by Chris Riccomini.

I have also added the Readthedocs badge for the latest version. Have created a 
[PR|https://github.com/apache/incubator-airflow/pull/3033].



was (Author: kaxilnaik):
I have updated the Pythonhosted links with the docs at 
http://airflow.incubator.apache.org/ . This has been documented at 
https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
 by Chris Riccomini.

I have also added the Readthedocs badge for the latest version. Have created a 
[PR|https://github.com/apache/incubator-airflow/pull/3033].

Regards,
Kaxil

> Links to the official docs are all broken
> -
>
> Key: AIRFLOW-2100
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs, Documentation
>Affects Versions: 1.9.0, 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.9.0, 2.0.0
>
>
> Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve 
> it:
> # Change the all the links to readthedocs 
> ([latest|http://airflow.readthedocs.io/en/latest/] / 
> [stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
> # Can someone who has access to PyPI create the docs?
> # Replace link to http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-2100:

Comment: was deleted

(was: [~Fokko] What do you suggest?)

> Links to the official docs are all broken
> -
>
> Key: AIRFLOW-2100
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs, Documentation
>Affects Versions: 1.9.0, 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.9.0, 2.0.0
>
>
> Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve 
> it:
> # Change the all the links to readthedocs 
> ([latest|http://airflow.readthedocs.io/en/latest/] / 
> [stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
> # Can someone who has access to PyPI create the docs?
> # Replace link to http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-2100:

Description: 
Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve it:

# Change the all the links to readthedocs 
([latest|http://airflow.readthedocs.io/en/latest/] / 
[stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
# Can someone who has access to PyPI create the docs?
# Replace link to http://airflow.incubator.apache.org/

  was:
Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve it:

# Change the all the links to readthedocs 
([latest|http://airflow.readthedocs.io/en/latest/] / 
[stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
# Can someone who has access to PyPI create the docs?


> Links to the official docs are all broken
> -
>
> Key: AIRFLOW-2100
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs, Documentation
>Affects Versions: 1.9.0, 2.0.0
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Major
> Fix For: 1.9.0, 2.0.0
>
>
> Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve 
> it:
> # Change the all the links to readthedocs 
> ([latest|http://airflow.readthedocs.io/en/latest/] / 
> [stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
> # Can someone who has access to PyPI create the docs?
> # Replace link to http://airflow.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360104#comment-16360104
 ] 

Kaxil Naik commented on AIRFLOW-2100:
-

[~Fokko] What do you suggest?

> Links to the official docs are all broken
> -
>
> Key: AIRFLOW-2100
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs, Documentation
>Affects Versions: 1.9.0, 2.0.0
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.9.0, 2.0.0
>
>
> Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve 
> it:
> # Change the all the links to readthedocs 
> ([latest|http://airflow.readthedocs.io/en/latest/] / 
> [stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
> # Can someone who has access to PyPI create the docs?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2100) Links to the official docs are all broken

2018-02-11 Thread Kaxil Naik (JIRA) 
Kaxil Naik created AIRFLOW-2100:
---

 Summary: Links to the official docs are all broken
 Key: AIRFLOW-2100
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2100
 Project: Apache Airflow
  Issue Type: Task
  Components: docs, Documentation
Affects Versions: 1.9.0, 2.0.0
Reporter: Kaxil Naik
 Fix For: 1.9.0, 2.0.0


Links to Pythonhosted are all broken. We can do 1 of the 2 things to resolve it:

# Change the all the links to readthedocs 
([latest|http://airflow.readthedocs.io/en/latest/] / 
[stable|http://airflow.readthedocs.io/en/v1-9-stable/])?
# Can someone who has access to PyPI create the docs?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-1404) Big Query Operator add max_billing_tier, create_disposition and flatten_results

2018-02-11 Thread Fokko Driesprong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-1404.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #3030
[https://github.com/apache/incubator-airflow/pull/3030]

> Big Query Operator add max_billing_tier, create_disposition and 
> flatten_results 
> 
>
> Key: AIRFLOW-1404
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1404
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: operators
>Affects Versions: Airflow 1.8
>Reporter: Alex Aslan
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 2.0.0
>
>
> I would like to have the option to set explicitly the params 
> max_billing_tier, flatten_results and create_disposition in the contrib big 
> query operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator

2018-02-11 Thread fokko 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master 69334fc44 -> a289497c8


[AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator

- Updated BQ hook `run_query` method to add
'flatten_results' & 'maximum_bytes_billed'
parameters
- Added the same in BQ Operator

Closes #3030 from kaxil/AIRFLOW-1404


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a289497c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a289497c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a289497c

Branch: refs/heads/master
Commit: a289497c807c238deb34bac54bbc4b28cbda9440
Parents: 69334fc
Author: Kaxil Naik 
Authored: Sun Feb 11 21:18:28 2018 +0100
Committer: Fokko Driesprong 
Committed: Sun Feb 11 21:18:28 2018 +0100

--
 airflow/contrib/hooks/bigquery_hook.py | 15 +++
 airflow/contrib/operators/bigquery_operator.py | 20 +++-
 2 files changed, 34 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a289497c/airflow/contrib/hooks/bigquery_hook.py
--
diff --git a/airflow/contrib/hooks/bigquery_hook.py 
b/airflow/contrib/hooks/bigquery_hook.py
index 220156b..ce7e2c3 100644
--- a/airflow/contrib/hooks/bigquery_hook.py
+++ b/airflow/contrib/hooks/bigquery_hook.py
@@ -465,8 +465,10 @@ class BigQueryBaseCursor(LoggingMixin):
   destination_dataset_table=False,
   write_disposition='WRITE_EMPTY',
   allow_large_results=False,
+  flatten_results=False,
   udf_config=False,
   maximum_billing_tier=None,
+  maximum_bytes_billed=None,
   create_disposition='CREATE_IF_NEEDED',
   query_params=None,
   schema_update_options=(),
@@ -488,12 +490,22 @@ class BigQueryBaseCursor(LoggingMixin):
 :type write_disposition: string
 :param allow_large_results: Whether to allow large results.
 :type allow_large_results: boolean
+:param flatten_results: If true and query uses legacy SQL dialect, 
flattens
+all nested and repeated fields in the query results. 
``allowLargeResults``
+must be true if this is set to false. For standard SQL queries, 
this
+flag is ignored and results are never flattened.
+:type flatten_results: boolean
 :param udf_config: The User Defined Function configuration for the 
query.
 See https://cloud.google.com/bigquery/user-defined-functions for 
details.
 :type udf_config: list
 :param maximum_billing_tier: Positive integer that serves as a
 multiplier of the basic price.
 :type maximum_billing_tier: integer
+:param maximum_bytes_billed: Limits the bytes billed for this job.
+Queries that will have bytes billed beyond this limit will fail
+(without incurring a charge). If unspecified, this will be
+set to your project default.
+:type maximum_bytes_billed: float
 :param create_disposition: Specifies whether the job is allowed to
 create new tables.
 :type create_disposition: string
@@ -528,6 +540,7 @@ class BigQueryBaseCursor(LoggingMixin):
 'query': bql,
 'useLegacySql': self.use_legacy_sql,
 'maximumBillingTier': maximum_billing_tier,
+'maximumBytesBilled': maximum_bytes_billed,
 'priority': priority
 }
 }
@@ -542,6 +555,8 @@ class BigQueryBaseCursor(LoggingMixin):
 configuration['query'].update({
 'allowLargeResults':
 allow_large_results,
+'flattenResults':
+flatten_results,
 'writeDisposition':
 write_disposition,
 'createDisposition':

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a289497c/airflow/contrib/operators/bigquery_operator.py
--
diff --git a/airflow/contrib/operators/bigquery_operator.py 
b/airflow/contrib/operators/bigquery_operator.py
index d43a65c..ef1ada0 100644
--- a/airflow/contrib/operators/bigquery_operator.py
+++ b/airflow/contrib/operators/bigquery_operator.py
@@ -38,6 +38,13 @@ class BigQueryOperator(BaseOperator):
 :param create_disposition: Specifies whether the job is allowed to create 
new tables.
 (default: 'CREATE_IF_NEEDED')
 :type create_disposition: string
+:param allow_large_results: Whether to allow large

[jira] [Commented] (AIRFLOW-1404) Big Query Operator add max_billing_tier, create_disposition and flatten_results

2018-02-11 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360096#comment-16360096
 ] 

ASF subversion and git services commented on AIRFLOW-1404:
--

Commit a289497c807c238deb34bac54bbc4b28cbda9440 in incubator-airflow's branch 
refs/heads/master from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a289497 ]

[AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator

- Updated BQ hook `run_query` method to add
'flatten_results' & 'maximum_bytes_billed'
parameters
- Added the same in BQ Operator

Closes #3030 from kaxil/AIRFLOW-1404


> Big Query Operator add max_billing_tier, create_disposition and 
> flatten_results 
> 
>
> Key: AIRFLOW-1404
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1404
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: operators
>Affects Versions: Airflow 1.8
>Reporter: Alex Aslan
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 2.0.0
>
>
> I would like to have the option to set explicitly the params 
> max_billing_tier, flatten_results and create_disposition in the contrib big 
> query operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-1404) Big Query Operator add max_billing_tier, create_disposition and flatten_results

2018-02-11 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360097#comment-16360097
 ] 

ASF subversion and git services commented on AIRFLOW-1404:
--

Commit a289497c807c238deb34bac54bbc4b28cbda9440 in incubator-airflow's branch 
refs/heads/master from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a289497 ]

[AIRFLOW-1404] Add 'flatten_results' & 'maximum_bytes_billed' to BQ Operator

- Updated BQ hook `run_query` method to add
'flatten_results' & 'maximum_bytes_billed'
parameters
- Added the same in BQ Operator

Closes #3030 from kaxil/AIRFLOW-1404


> Big Query Operator add max_billing_tier, create_disposition and 
> flatten_results 
> 
>
> Key: AIRFLOW-1404
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1404
> Project: Apache Airflow
>  Issue Type: Wish
>  Components: operators
>Affects Versions: Airflow 1.8
>Reporter: Alex Aslan
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 2.0.0
>
>
> I would like to have the option to set explicitly the params 
> max_billing_tier, flatten_results and create_disposition in the contrib big 
> query operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-800) Initialize default Google BigQuery Connection with valid conn_type

2018-02-11 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360093#comment-16360093
 ] 

ASF subversion and git services commented on AIRFLOW-800:
-

Commit 69334fc44726784f2351ef5ecc9ce8be18e88939 in incubator-airflow's branch 
refs/heads/master from [~kaxilnaik]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=69334fc ]

[AIRFLOW-800] Initialize valid Google BigQuery Connection

- Modified `GoogleCloudBaseHook` to change the
name of parameter `conn_id` to keep it consistent
with other Hooks.
- Changed the connection to `GoogleCloudBaseHook`
instead of `BigQueryHook` which was causing an
invalid `conn_type` creation during `airflow
initdb`

Closes #3031 from kaxil/AIRFLOW-800


> Initialize default Google BigQuery Connection with valid conn_type
> --
>
> Key: AIRFLOW-800
> URL: https://issues.apache.org/jira/browse/AIRFLOW-800
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Reporter: Wilson Lian
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> {{airflow initdb}} creates a connection with conn_id='bigquery_default' and 
> conn_type='bigquery'. However, bigquery is not a valid conn_type, according 
> to models.Connection._types, and BigQuery connections should use the 
> google_cloud_platform conn_type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AIRFLOW-800) Initialize default Google BigQuery Connection with valid conn_type

2018-02-11 Thread Fokko Driesprong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AIRFLOW-800.
--
   Resolution: Fixed
Fix Version/s: 2.0.0

Issue resolved by pull request #3031
[https://github.com/apache/incubator-airflow/pull/3031]

> Initialize default Google BigQuery Connection with valid conn_type
> --
>
> Key: AIRFLOW-800
> URL: https://issues.apache.org/jira/browse/AIRFLOW-800
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: utils
>Reporter: Wilson Lian
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 2.0.0
>
>
> {{airflow initdb}} creates a connection with conn_id='bigquery_default' and 
> conn_type='bigquery'. However, bigquery is not a valid conn_type, according 
> to models.Connection._types, and BigQuery connections should use the 
> google_cloud_platform conn_type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

incubator-airflow git commit: [AIRFLOW-800] Initialize valid Google BigQuery Connection

2018-02-11 Thread fokko 
Repository: incubator-airflow
Updated Branches:
  refs/heads/master fe73f2215 -> 69334fc44


[AIRFLOW-800] Initialize valid Google BigQuery Connection

- Modified `GoogleCloudBaseHook` to change the
name of parameter `conn_id` to keep it consistent
with other Hooks.
- Changed the connection to `GoogleCloudBaseHook`
instead of `BigQueryHook` which was causing an
invalid `conn_type` creation during `airflow
initdb`

Closes #3031 from kaxil/AIRFLOW-800


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/69334fc4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/69334fc4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/69334fc4

Branch: refs/heads/master
Commit: 69334fc44726784f2351ef5ecc9ce8be18e88939
Parents: fe73f22
Author: Kaxil Naik 
Authored: Sun Feb 11 21:16:24 2018 +0100
Committer: Fokko Driesprong 
Committed: Sun Feb 11 21:16:28 2018 +0100

--
 airflow/contrib/hooks/bigquery_hook.py |  2 +-
 airflow/contrib/hooks/gcp_api_base_hook.py | 19 ++-
 airflow/models.py  |  4 ++--
 3 files changed, 13 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/69334fc4/airflow/contrib/hooks/bigquery_hook.py
--
diff --git a/airflow/contrib/hooks/bigquery_hook.py 
b/airflow/contrib/hooks/bigquery_hook.py
index dca4d33..220156b 100644
--- a/airflow/contrib/hooks/bigquery_hook.py
+++ b/airflow/contrib/hooks/bigquery_hook.py
@@ -49,7 +49,7 @@ class BigQueryHook(GoogleCloudBaseHook, DbApiHook, 
LoggingMixin):
  delegate_to=None,
  use_legacy_sql=True):
 super(BigQueryHook, self).__init__(
-conn_id=bigquery_conn_id, delegate_to=delegate_to)
+gcp_conn_id=bigquery_conn_id, delegate_to=delegate_to)
 self.use_legacy_sql = use_legacy_sql
 
 def get_conn(self):

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/69334fc4/airflow/contrib/hooks/gcp_api_base_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_api_base_hook.py 
b/airflow/contrib/hooks/gcp_api_base_hook.py
index e6ca240..12a7fb5 100644
--- a/airflow/contrib/hooks/gcp_api_base_hook.py
+++ b/airflow/contrib/hooks/gcp_api_base_hook.py
@@ -43,18 +43,19 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
 
 Legacy P12 key files are not supported.
 """
-def __init__(self, conn_id, delegate_to=None):
+
+def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
 """
-:param conn_id: The connection ID to use when fetching connection info.
-:type conn_id: string
+:param gcp_conn_id: The connection ID to use when fetching connection 
info.
+:type gcp_conn_id: string
 :param delegate_to: The account to impersonate, if any.
 For this to work, the service account making the request must have
 domain-wide delegation enabled.
 :type delegate_to: string
 """
-self.conn_id = conn_id
+self.gcp_conn_id = gcp_conn_id
 self.delegate_to = delegate_to
-self.extras = self.get_connection(conn_id).extra_dejson
+self.extras = self.get_connection(self.gcp_conn_id).extra_dejson
 
 def _get_credentials(self):
 """
@@ -69,8 +70,8 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
 kwargs['sub'] = self.delegate_to
 
 if not key_path and not keyfile_dict:
-self.log.info('Getting connection using `gcloud auth` user, since 
no key file '
- 'is defined for hook.')
+self.log.info('Getting connection using `gcloud auth` user, '
+  'since no key file is defined for hook.')
 credentials = GoogleCredentials.get_application_default()
 elif key_path:
 if not scope:
@@ -80,7 +81,7 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
 # Get credentials from a JSON file.
 if key_path.endswith('.json'):
 self.log.info('Getting connection using a JSON key file.')
-credentials = ServiceAccountCredentials\
+credentials = ServiceAccountCredentials \
 .from_json_keyfile_name(key_path, scopes)
 elif key_path.endswith('.p12'):
 raise AirflowException('Legacy P12 key file are not supported, 
'
@@ -101,7 +102,7 @@ class GoogleCloudBaseHook(BaseHook, LoggingMixin):
 keyfile_dict['private_key'] = 
keyfile_dict['private_key'].replace(
 '\\n', '\n')
 
-

[jira] [Commented] (AIRFLOW-2052) Update documentation to reflect the correct PyPI package

2018-02-11 Thread Kaxil Naik (JIRA) 

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360088#comment-16360088
 ] 

Kaxil Naik commented on AIRFLOW-2052:
-

Which documentation are you referring to?

Can you please provide the link to the one you are referring to?

> Update documentation to reflect the correct PyPI package
> 
>
> Key: AIRFLOW-2052
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2052
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.8.1, 1.9.0, 1.8.2
>Reporter: Michele De Simoni
>Priority: Major
> Fix For: 1.8.1, 1.9.0, 1.8.2
>
>
> The Documentation suggests installing using {code:bash}pip install 
> airflow{code}
> However this would only get the 1.8.0 version. Correct behavior should be to 
> {code:bash}pip install apache-airflow{code} as stated also on the PyPI site. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (AIRFLOW-2099) Task details cannot be shown when PythonOperator calls partial function / class instance with __call__

2018-02-11 Thread Matthew Revell (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-2099 started by Matthew Revell.
---
> Task details cannot be shown when PythonOperator calls partial function / 
> class instance with __call__
> --
>
> Key: AIRFLOW-2099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2099
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: ui
>Affects Versions: Airflow 1.8
>Reporter: Matthew Revell
>Assignee: Matthew Revell
>Priority: Minor
>
> There are several scenarios where the inspect.getsource() method fails with:
> {{object at 0x> is not a module, class, method, function, traceback, 
> frame, or code object}}
> One such scenario is described in 
> [AIRFLOW-1027|https://issues.apache.org/jira/browse/AIRFLOW-1027] where a 
> partial function is used. Another is when an instance of a class which 
> implements __call__() is used.
> Example:
> {{class MyClass(object):}}
> {{    def __init__(self):}}
> {{        pass}}
> {{    def __call__(self):}}
> {{        pass}}
> {{my_class = MyClass()}}
> {{dag_task = PythonOperator(}}
> {{    task_id='dag_task',}}
> {{    dag=dag, }}
> {{    python_callable=my_class,}}
> {{)}}
> There exists a PR for AIRFLOW-1027, however, this fix does not address this 
> other scenario, and also does not guard against any other edge cases which my 
> result in this error in future.
> A better solution would be to catch known scenarios with work arounds, and 
> default to reporting that the source is unavailable for unknown cases. This 
> would at least display the Task Instance details in every case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-2099) Task details cannot be shown when PythonOperator calls partial function / class instance with __call__

2018-02-11 Thread Matthew Revell (JIRA) 
Matthew Revell created AIRFLOW-2099:
---

 Summary: Task details cannot be shown when PythonOperator calls 
partial function / class instance with __call__
 Key: AIRFLOW-2099
 URL: https://issues.apache.org/jira/browse/AIRFLOW-2099
 Project: Apache Airflow
  Issue Type: Bug
  Components: ui
Affects Versions: Airflow 1.8
Reporter: Matthew Revell
Assignee: Matthew Revell


There are several scenarios where the inspect.getsource() method fails with:

{{object at 0x> is not a module, class, method, function, traceback, 
frame, or code object}}

One such scenario is described in 
[AIRFLOW-1027|https://issues.apache.org/jira/browse/AIRFLOW-1027] where a 
partial function is used. Another is when an instance of a class which 
implements __call__() is used.

Example:

{{class MyClass(object):}}
{{    def __init__(self):}}
{{        pass}}

{{    def __call__(self):}}
{{        pass}}
{{my_class = MyClass()}}
{{dag_task = PythonOperator(}}
{{    task_id='dag_task',}}
{{    dag=dag, }}
{{    python_callable=my_class,}}
{{)}}

There exists a PR for AIRFLOW-1027, however, this fix does not address this 
other scenario, and also does not guard against any other edge cases which my 
result in this error in future.

A better solution would be to catch known scenarios with work arounds, and 
default to reporting that the source is unavailable for unknown cases. This 
would at least display the Task Instance details in every case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-2098) TimeDeltaSensor task consume cpu too high

2018-02-11 Thread luoshaohua (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luoshaohua updated AIRFLOW-2098:

Description: why each TimeDeltaSensor task consume cpu 6%? i think those 
tasks is a simple sleep, so consume too high.   (was: why each TimeDeltaSensor 
task consume cpu 6%, i think those tasks is a simple sleep, so consume too 
high. )

> TimeDeltaSensor task consume cpu too high
> -
>
> Key: AIRFLOW-2098
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2098
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Affects Versions: Airflow 1.8
>Reporter: luoshaohua
>Priority: Major
> Attachments: QQ截图20180211154639.png
>
>
> why each TimeDeltaSensor task consume cpu 6%? i think those tasks is a simple 
> sleep, so consume too high. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)