[jira] [Updated] (AIRFLOW-3171) Flexible task log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the task logs in a more flexible manner, rather than 
defaulting to a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the task 
logs in s3 were organized like /tasks/[owner]/... then we could provide access 
to a subset of the logs for each team, by creating s3 access rules prefixed 
with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the task logs in a more flexible manner, rather than 
defaulting to a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.


> Flexible task log organization
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the task logs in a more flexible manner, rather than 
> defaulting to a flat structure.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. For example, if the 
> task logs in s3 were organized like /tasks/[owner]/... then we could provide 
> access to a subset of the logs for each team, by creating s3 access rules 
> prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Flexible task log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the task logs in a more flexible manner, rather than 
defaulting to a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs in a more flexible manner, rather than defaulting to 
a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.


> Flexible task log organization
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the task logs in a more flexible manner, rather than 
> defaulting to a flat structure.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. For example, if the 
> log folders in s3 were organized like /tasks/[owner]/... then we could 
> provide access to a subset of the logs for each team, by creating s3 access 
> rules prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Flexible log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs in a more flexible manner, rather than defaulting to 
a flat structure.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs in a more flexible manner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.


> Flexible log organization
> -
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs in a more flexible manner, rather than defaulting 
> to a flat structure.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. For example, if the 
> log folders in s3 were organized like /tasks/[owner]/... then we could 
> provide access to a subset of the logs for each team, by creating s3 access 
> rules prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Flexible task log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Summary: Flexible task log organization  (was: Flexible log organization)

> Flexible task log organization
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs in a more flexible manner, rather than defaulting 
> to a flat structure.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. For example, if the 
> log folders in s3 were organized like /tasks/[owner]/... then we could 
> provide access to a subset of the logs for each team, by creating s3 access 
> rules prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Flexible log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs in a more flexible manner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. For example, if the log 
folders in s3 were organized like /tasks/[owner]/... then we could provide 
access to a subset of the logs for each team, by creating s3 access rules 
prefixed with the appropriate path.

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.


> Flexible log organization
> -
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs in a more flexible manner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. For example, if the 
> log folders in s3 were organized like /tasks/[owner]/... then we could 
> provide access to a subset of the logs for each team, by creating s3 access 
> rules prefixed with the appropriate path.
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Flexible log organization

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Summary: Flexible log organization  (was: Organize logs by owner)

> Flexible log organization
> -
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs by owner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. If the log folders 
> in s3 were organized like /tasks/[owner]/... then we could provide access to 
> a subset of the logs for each team, by creating s3 access rules prefixed with 
> the appropriate path.
>  
> I also think that this would be a good change regardless of the multi-tenancy 
> aspect, just in terms of organizing logs (vs the current flat namespace).
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Organize logs by owner

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

 

One possible implementation would be to have a configurable, templatized path 
structure for logs. We would also need to store the log location for each task 
instance, so we could easily change the log folder structure without breaking 
old log paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

 

One possible implementation would be to have a configurable, templatized path 
structure. We would also need to store the log location for each task instance, 
so we could easily change the log folder structure without breaking old log 
paths.


> Organize logs by owner
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs by owner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. If the log folders 
> in s3 were organized like /tasks/[owner]/... then we could provide access to 
> a subset of the logs for each team, by creating s3 access rules prefixed with 
> the appropriate path.
>  
> I also think that this would be a good change regardless of the multi-tenancy 
> aspect, just in terms of organizing logs (vs the current flat namespace).
>  
> One possible implementation would be to have a configurable, templatized path 
> structure for logs. We would also need to store the log location for each 
> task instance, so we could easily change the log folder structure without 
> breaking old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Organize logs by owner

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

 

One possible implementation would be to have a configurable, templatized path 
structure. We would also need to store the log location for each task instance, 
so we could easily change the log folder structure without breaking old log 
paths.

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).


> Organize logs by owner
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs by owner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. If the log folders 
> in s3 were organized like /tasks/[owner]/... then we could provide access to 
> a subset of the logs for each team, by creating s3 access rules prefixed with 
> the appropriate path.
>  
> I also think that this would be a good change regardless of the multi-tenancy 
> aspect, just in terms of organizing logs (vs the current flat namespace).
>  
> One possible implementation would be to have a configurable, templatized path 
> structure. We would also need to store the log location for each task 
> instance, so we could easily change the log folder structure without breaking 
> old log paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Organize logs by owner

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the owner 
string.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each time, by creating s3 access rules prefixed with the owner 
string.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).


> Organize logs by owner
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs by owner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. If the log folders 
> in s3 were organized like /tasks/[owner]/... then we could provide access to 
> a subset of the logs for each team, by creating s3 access rules prefixed with 
> the owner string.
>  
> I also think that this would be a good change regardless of the multi-tenancy 
> aspect, just in terms of organizing logs (vs the current flat namespace).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3171) Organize logs by owner

[Gabriel Silk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Silk updated AIRFLOW-3171:
--
Description: 
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the 
appropriate path.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).

  was:
Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each team, by creating s3 access rules prefixed with the owner 
string.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).


> Organize logs by owner
> --
>
> Key: AIRFLOW-3171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: logging
>Reporter: Gabriel Silk
>Priority: Minor
>
> Regardless of the backend (eg file system, s3, ...), it would be useful to be 
> able to organize the logs by owner.
>  
> One use case of this would be to provide a better multi-tenancy experience 
> when deploying a single airflow cluster to several teams. If the log folders 
> in s3 were organized like /tasks/[owner]/... then we could provide access to 
> a subset of the logs for each team, by creating s3 access rules prefixed with 
> the appropriate path.
>  
> I also think that this would be a good change regardless of the multi-tenancy 
> aspect, just in terms of organizing logs (vs the current flat namespace).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3171) Organize logs by owner

[Gabriel Silk (JIRA)]

Gabriel Silk created AIRFLOW-3171:
-

 Summary: Organize logs by owner
 Key: AIRFLOW-3171
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3171
 Project: Apache Airflow
  Issue Type: Improvement
  Components: logging
Reporter: Gabriel Silk


Regardless of the backend (eg file system, s3, ...), it would be useful to be 
able to organize the logs by owner.

 

One use case of this would be to provide a better multi-tenancy experience when 
deploying a single airflow cluster to several teams. If the log folders in s3 
were organized like /tasks/[owner]/... then we could provide access to a subset 
of the logs for each time, by creating s3 access rules prefixed with the owner 
string.

 

I also think that this would be a good change regardless of the multi-tenancy 
aspect, just in terms of organizing logs (vs the current flat namespace).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3078) Basic operators for Google Compute Engine

[Jarek Potiuk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jarek Potiuk updated AIRFLOW-3078:
--
Description: 
In order to be able to interact with raw Google Compute Engine, we need an 
operator that should be able to:

For managing individual machines:
 * Start Instance: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/start])
 * Set Machine Type 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType])
 
 * Stop Instance: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop])

 

  was:
In order to be able to interact with raw Google Compute Engine, we need an 
operator that should be able to:

For managing individual machines:
 * Start Instance: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/start])
 * Set Machine Type 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType])
 
 * Stop Instance: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop])

Also we should be able to manipulate instance groups:
 * Get instance group: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/get])
 * Insert Group: 
([https://cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/insert])
 * Update Group: 
([https://cloud.google.com/compute/docs/reference/rest/beta/instanceGroupManagers/update])

 


> Basic operators for Google Compute Engine
> -
>
> Key: AIRFLOW-3078
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3078
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, gcp
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Trivial
>
> In order to be able to interact with raw Google Compute Engine, we need an 
> operator that should be able to:
> For managing individual machines:
>  * Start Instance: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/start])
>  * Set Machine Type 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType])
>  
>  * Stop Instance: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop])
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io commented on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

codecov-io commented on issue #4005: [AIRFLOW-3160] Load latest_dagruns 
asynchronously
URL: 
https://github.com/apache/incubator-airflow/pull/4005#issuecomment-427952710
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=h1)
 Report
   > Merging 
[#4005](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **decrease** coverage by `0.04%`.
   > The diff coverage is `57.69%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4005/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4005  +/-   ##
   ==
   - Coverage   75.79%   75.74%   -0.05% 
   ==
 Files 199  199  
 Lines   1594615972  +26 
   ==
   + Hits1208612098  +12 
   - Misses   3860 3874  +14
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.06% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.54% <26.66%> (-0.5%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=footer).
 Last update 
[b8be322...6607e48](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

codecov-io edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns 
asynchronously
URL: 
https://github.com/apache/incubator-airflow/pull/4005#issuecomment-427508156
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=h1)
 Report
   > Merging 
[#4005](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **decrease** coverage by `0.04%`.
   > The diff coverage is `57.69%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4005/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4005  +/-   ##
   ==
   - Coverage   75.79%   75.74%   -0.05% 
   ==
 Files 199  199  
 Lines   1594615972  +26 
   ==
   + Hits1208612098  +12 
   - Misses   3860 3874  +14
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.06% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.54% <26.66%> (-0.5%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=footer).
 Last update 
[b8be322...6607e48](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

codecov-io edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns 
asynchronously
URL: 
https://github.com/apache/incubator-airflow/pull/4005#issuecomment-427508156
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=h1)
 Report
   > Merging 
[#4005](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **decrease** coverage by `0.04%`.
   > The diff coverage is `57.69%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4005/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4005  +/-   ##
   ==
   - Coverage   75.79%   75.74%   -0.05% 
   ==
 Files 199  199  
 Lines   1594615972  +26 
   ==
   + Hits1208612098  +12 
   - Misses   3860 3874  +14
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.06% <100%> (+0.2%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `71.54% <26.66%> (-0.5%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4005/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=footer).
 Last update 
[b8be322...6607e48](https://codecov.io/gh/apache/incubator-airflow/pull/4005?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Eronarn commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism

[GitBox]

Eronarn commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism
URL: 
https://github.com/apache/incubator-airflow/pull/3584#issuecomment-427929546
 
 
   Yes I do plan to work on this again at some point. But it'll need rebasing 
and such, might be a while.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4022: [AIRFLOW-3078] Basic operators for Google Compute Engine

[GitBox]

codecov-io edited a comment on issue #4022: [AIRFLOW-3078] Basic operators for 
Google Compute Engine
URL: 
https://github.com/apache/incubator-airflow/pull/4022#issuecomment-427925561
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=h1)
 Report
   > Merging 
[#4022](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4022/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4022  +/-   ##
   ==
   - Coverage   75.79%   75.76%   -0.03% 
   ==
 Files 199  199  
 Lines   1594615946  
   ==
   - Hits1208612082   -4 
   - Misses   3860 3864   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4022/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4022/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.67% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=footer).
 Last update 
[b8be322...7005dd7](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4022: [AIRFLOW-3078] Basic operators for Google Compute Engine

[GitBox]

codecov-io commented on issue #4022: [AIRFLOW-3078] Basic operators for Google 
Compute Engine
URL: 
https://github.com/apache/incubator-airflow/pull/4022#issuecomment-427925561
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=h1)
 Report
   > Merging 
[#4022](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4022/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4022  +/-   ##
   ==
   - Coverage   75.79%   75.76%   -0.03% 
   ==
 Files 199  199  
 Lines   1594615946  
   ==
   - Hits1208612082   -4 
   - Misses   3860 3864   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4022/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4022/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.67% <0%> (-0.05%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=footer).
 Last update 
[b8be322...7005dd7](https://codecov.io/gh/apache/incubator-airflow/pull/4022?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added

[GitBox]

phani8996 commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for 
airflow added
URL: 
https://github.com/apache/incubator-airflow/pull/3989#issuecomment-427924698
 
 
   > @phani8996 whats harm in letting my machine run with the full capacity of 
workers all the time? Cause if I am allowing to grow it to a max, then it means 
that my machine has the capacity to handle that many workers anyway.
   
   We can run it at full capacity, but what advantage are we going to get with 
a bunch of idle workers? Instead this feature spawns workers as per demand. In 
a way you get what is required. No more under utilisation of workers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log

[GitBox]

phani8996 edited a comment on issue #3992: [AIRFLOW-620] Feature to tail custom 
number of logs instead of rendering whole log
URL: 
https://github.com/apache/incubator-airflow/pull/3992#issuecomment-427917299
 
 
   > @phani8996 can you post some gifs on this feature in work?
   
   @msumit I am not able to upload gif. I am attaching images for reference.
   
   
![img-20181008-wa0013](https://user-images.githubusercontent.com/12140904/46624506-99061380-cb4e-11e8-9005-b14031280b4c.jpg)
   
![img-20181008-wa0005](https://user-images.githubusercontent.com/12140904/46624523-a8855c80-cb4e-11e8-9d05-25a82bd951ee.jpg)
   
![img-20181008-wa0004](https://user-images.githubusercontent.com/12140904/46624529-ade2a700-cb4e-11e8-9562-279dd2823eb9.jpg)
   
![img-20181008-wa0003](https://user-images.githubusercontent.com/12140904/46624560-c2bf3a80-cb4e-11e8-9737-aef1dc2879a7.jpg)
   
![img-20181008-wa0002](https://user-images.githubusercontent.com/12140904/46624571-cbb00c00-cb4e-11e8-8809-8faa958c3866.jpg)
   
   
   
   
   
   
   
   
   
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] phani8996 commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log

[GitBox]

phani8996 commented on issue #3992: [AIRFLOW-620] Feature to tail custom number 
of logs instead of rendering whole log
URL: 
https://github.com/apache/incubator-airflow/pull/3992#issuecomment-427917299
 
 
   
   
   
   
   
   
   > @phani8996 can you post some gifs on this feature in work?
   
   @msumit I am not able to upload gif. I am attaching images for reference.
   
   
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3078) Basic operators for Google Compute Engine

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642197#comment-16642197
 ] 

ASF GitHub Bot commented on AIRFLOW-3078:
-

sprzedwojski opened a new pull request #4022: [AIRFLOW-3078] Basic operators 
for Google Compute Engine
URL: https://github.com/apache/incubator-airflow/pull/4022
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3078) issue and references 
it in the PR title.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Add GceInstanceStartOperator, GceInstanceStopOperator and 
GceSetMachineTypeOperator.
   
   Each operator includes:
   - core logic
   - input params validation
   - unit tests
   - presence in the example DAG
   - docstrings
   - How-to and Integration documentation
   
   Additionally, in GceHook error checking if response is 200 OK was added:
   
   Some types of errors are only visible in the response's "error" field
   and the overall HTTP response is 200 OK.
   
   That is why apart from checking if status is "done" we also check
   if "error" is empty, and if not an exception is raised with error
   message extracted from the "error" field of the response.
   
   In this commit we also separated out Body Field Validator to
   separate module in tools - this way it can be reused between
   various GCP operators, it has proven to be usable in at least
   two of them now.
   
   ### Tests
   
   - [x] My PR adds the following unit tests:
   `test_gcp_function_operator`
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Basic operators for Google Compute Engine
> -
>
> Key: AIRFLOW-3078
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3078
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, gcp
>Reporter: Jarek Potiuk
>Assignee: Jarek Potiuk
>Priority: Trivial
>
> In order to be able to interact with raw Google Compute Engine, we need an 
> operator that should be able to:
> For managing individual machines:
>  * Start Instance: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/start])
>  * Set Machine Type 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/setMachineType])
>  
>  * Stop Instance: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop])
> Also we should be able to manipulate instance groups:
>  * Get instance group: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/get])
>  * Insert Group: 
> ([https://cloud.google.com/compute/docs/reference/rest/v1/instanceGroupManagers/insert])
>  * Update Group: 
> ([https://cloud.google.com/compute/docs/reference/rest/beta/instanceGroupManagers/update])
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] sprzedwojski opened a new pull request #4022: [AIRFLOW-3078] Basic operators for Google Compute Engine

[GitBox]

sprzedwojski opened a new pull request #4022: [AIRFLOW-3078] Basic operators 
for Google Compute Engine
URL: https://github.com/apache/incubator-airflow/pull/4022
 
 
   ### Jira
   
   - [x] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3078) issue and references 
it in the PR title.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   Add GceInstanceStartOperator, GceInstanceStopOperator and 
GceSetMachineTypeOperator.
   
   Each operator includes:
   - core logic
   - input params validation
   - unit tests
   - presence in the example DAG
   - docstrings
   - How-to and Integration documentation
   
   Additionally, in GceHook error checking if response is 200 OK was added:
   
   Some types of errors are only visible in the response's "error" field
   and the overall HTTP response is 200 OK.
   
   That is why apart from checking if status is "done" we also check
   if "error" is empty, and if not an exception is raised with error
   message extracted from the "error" field of the response.
   
   In this commit we also separated out Body Field Validator to
   separate module in tools - this way it can be reused between
   various GCP operators, it has proven to be usable in at least
   two of them now.
   
   ### Tests
   
   - [x] My PR adds the following unit tests:
   `test_gcp_function_operator`
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sganz opened a new pull request #4021: [AIRFLOW-XXX] - Added CarLabs to users

[GitBox]

sganz opened a new pull request #4021: [AIRFLOW-XXX] - Added CarLabs to users
URL: https://github.com/apache/incubator-airflow/pull/4021
 
 
   Been using it over a year now. Great work!
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   Added CarLabs to list of users
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

[Aaron Dossett (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642117#comment-16642117
 ] 

Aaron Dossett commented on AIRFLOW-2516:


Could this be closed a duplicate of 2511 to consolidate discussion in that 
issue?

> Deadlock found when trying to update task_instance table
> 
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
>
>  
>  
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: 
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: 
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227 
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback 
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/bin/airflow", line 27, in 
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in 
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = 
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in 
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in 
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: 
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = 
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, 
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, 
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, 
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, 
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, 
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, 
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, 
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return 
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, 
> in _execute_and_instances
> [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: 
> close_with_result=True)
> [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: File 
> 

[GitHub] codecov-io edited a comment on issue #3885: [AIRFLOW-3001] Add index 'ti_dag_date' to taskinstance

[GitBox]

codecov-io edited a comment on issue #3885: [AIRFLOW-3001] Add index 
'ti_dag_date' to taskinstance
URL: 
https://github.com/apache/incubator-airflow/pull/3885#issuecomment-427268064
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=h1)
 Report
   > Merging 
[#3885](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc)
 will **increase** coverage by `60.27%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3885/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master   #3885   +/-   ##
   ==
   + Coverage   15.22%   75.5%   +60.27% 
   ==
 Files 199 199   
 Lines   15946   15947+1 
   ==
   + Hits 2428   12040 +9612 
   + Misses  135183907 -9611
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.71% <ø> (+63.94%)` | :arrow_up: |
   | 
[airflow/exceptions.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5)
 | `100% <0%> (+2.85%)` | :arrow_up: |
   | 
[airflow/utils/operator\_resources.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9vcGVyYXRvcl9yZXNvdXJjZXMucHk=)
 | `86.95% <0%> (+4.34%)` | :arrow_up: |
   | 
[airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=)
 | `55.76% <0%> (+5.76%)` | :arrow_up: |
   | 
[airflow/utils/decorators.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5)
 | `91.66% <0%> (+14.58%)` | :arrow_up: |
   | 
[airflow/settings.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==)
 | `81.15% <0%> (+15.21%)` | :arrow_up: |
   | 
[airflow/hooks/oracle\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9vcmFjbGVfaG9vay5weQ==)
 | `15.47% <0%> (+15.47%)` | :arrow_up: |
   | 
[airflow/task/task\_runner/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL19faW5pdF9fLnB5)
 | `63.63% <0%> (+18.18%)` | :arrow_up: |
   | 
[airflow/utils/db.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==)
 | `33.6% <0%> (+18.4%)` | :arrow_up: |
   | 
[airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==)
 | `74.28% <0%> (+19.99%)` | :arrow_up: |
   | ... and [151 
more](https://codecov.io/gh/apache/incubator-airflow/pull/3885/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=footer).
 Last update 
[b8be322...4090057](https://codecov.io/gh/apache/incubator-airflow/pull/3885?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3156) AWS Batch executor

[Hugo Prudente (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642085#comment-16642085
 ] 

Hugo Prudente commented on AIRFLOW-3156:


Hi Jacopo can you clarify a little bit more about your use case and the 
difference of using the Executor and the Operator for you?

> AWS Batch executor
> --
>
> Key: AIRFLOW-3156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3156
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: executor
>Reporter: Jacopo Sabbatini
>Priority: Major
>  Labels: aws
>
> Add support for AWS Batch as executor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] r39132 closed pull request #4020: [Airflow-XXX] Add MySQL connection to the docs

[GitBox]

r39132 closed pull request #4020:  [Airflow-XXX] Add MySQL connection to the 
docs 
URL: https://github.com/apache/incubator-airflow/pull/4020
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/howto/manage-connections.rst 
b/docs/howto/manage-connections.rst
index f869a08b3c..a79fe0bc30 100644
--- a/docs/howto/manage-connections.rst
+++ b/docs/howto/manage-connections.rst
@@ -133,3 +133,26 @@ Scopes (comma separated)
 Scopes are ignored when using application default credentials. See
 issue `AIRFLOW-2522
 `_.
+MySQL
+~
+The MySQL connect type allows to connect with MySQL database.
+
+Configuring the Connection
+''
+Host (required)
+The host to connect to.
+
+Schema (optional)
+Specify the schema name to be used in the database.
+
+Login (required)
+Specify the user name to connect.
+
+Password (required)
+Specify the password to connect.
+
+Extra (optional)
+Specify the charset. Example: {"charset": "utf8"}
+
+.. note::
+If encounter UnicodeDecodeError while working with MySQL connection 
check the charset defined is matched to the database charset.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4020: [Airflow-XXX] Add MySQL connection to the docs

[GitBox]

codecov-io commented on issue #4020:  [Airflow-XXX] Add MySQL connection to the 
docs 
URL: 
https://github.com/apache/incubator-airflow/pull/4020#issuecomment-427888671
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=h1)
 Report
   > Merging 
[#4020](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/683dc19fe36afc0f653bdd1d10a08918d48be335?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4020/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4020  +/-   ##
   ==
   + Coverage   75.76%   75.77%   +<.01% 
   ==
 Files 199  199  
 Lines   1594615946  
   ==
   + Hits1208212083   +1 
   + Misses   3864 3863   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4020/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.71% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=footer).
 Last update 
[683dc19...7d5c849](https://codecov.io/gh/apache/incubator-airflow/pull/4020?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] r39132 closed pull request #4019: Included Danamica in list of companies using Airflow

[GitBox]

r39132 closed pull request #4019: Included Danamica in list of companies using 
Airflow
URL: https://github.com/apache/incubator-airflow/pull/4019
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 9d1da1e49c..edeb46f6c5 100644
--- a/README.md
+++ b/README.md
@@ -144,6 +144,7 @@ Currently **officially** using Airflow:
 1. [Creditas](https://www.creditas.com.br) 
[[@dcassiano](https://github.com/dcassiano)]
 1. [Custom Ink](https://www.customink.com/) 
[[@david-dalisay](https://github.com/david-dalisay), 
[@dmartin11](https://github.com/dmartin11) & 
[@mpeteuil](https://github.com/mpeteuil)]
 1. [Dailymotion](http://www.dailymotion.com/fr) 
[[@germaintanguy](https://github.com/germaintanguy) & 
[@hc](https://github.com/hc)]
+1. [Danamica](https://www.danamica.dk) 
[[@testvinder](https://github.com/testvinder)]
 1. [Data Reply](https://www.datareply.co.uk/) 
[[@kaxil](https://github.com/kaxil)]
 1. [DataCamp](https://datacamp.com/) [[@dgrtwo](https://github.com/dgrtwo)]
 1. [DataFox](https://www.datafox.com/) 
[[@sudowork](https://github.com/sudowork)]


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] r39132 commented on issue #4019: Included Danamica in list of companies using Airflow

[GitBox]

r39132 commented on issue #4019: Included Danamica in list of companies using 
Airflow
URL: 
https://github.com/apache/incubator-airflow/pull/4019#issuecomment-427888044
 
 
   @testvinder Welcome aboard!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ron819 opened a new pull request #4020: [Airflow-XXX] Add MySQL connection to the docs

[GitBox]

ron819 opened a new pull request #4020:  [Airflow-XXX] Add MySQL connection to 
the docs 
URL: https://github.com/apache/incubator-airflow/pull/4020
 
 
   Explain how to connect with MySQL
   Add the example here: 
https://stackoverflow.com/questions/46084744/how-to-explicitly-declare-charset-utf8-for-airflow-connections
  to the documentation.
   
   Also explain how to avoid error as described here:
   
https://stackoverflow.com/questions/52699046/mysqltogooglecloudstorageoperator-fails-unexpectadly


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4019: Included Danamica in list of companies using Airflow

[GitBox]

codecov-io commented on issue #4019: Included Danamica in list of companies 
using Airflow
URL: 
https://github.com/apache/incubator-airflow/pull/4019#issuecomment-427876443
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=h1)
 Report
   > Merging 
[#4019](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/683dc19fe36afc0f653bdd1d10a08918d48be335?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4019/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4019  +/-   ##
   ==
   + Coverage   75.76%   75.77%   +<.01% 
   ==
 Files 199  199  
 Lines   1594615946  
   ==
   + Hits1208212083   +1 
   + Misses   3864 3863   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4019/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=)
 | `91.71% <0%> (+0.04%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=footer).
 Last update 
[683dc19...66ca358](https://codecov.io/gh/apache/incubator-airflow/pull/4019?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4018: [AIRFLOW-XXX] Log query and task_id in druid-hook

[GitBox]

codecov-io edited a comment on issue #4018: [AIRFLOW-XXX] Log query and task_id 
in druid-hook
URL: 
https://github.com/apache/incubator-airflow/pull/4018#issuecomment-427872750
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=h1)
 Report
   > Merging 
[#4018](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/683dc19fe36afc0f653bdd1d10a08918d48be335?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4018/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4018  +/-   ##
   ==
   + Coverage   75.76%   75.77%   +<.01% 
   ==
 Files 199  199  
 Lines   1594615948   +2 
   ==
   + Hits1208212084   +2 
 Misses   3864 3864
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4018/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5)
 | `88% <100%> (+0.32%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=footer).
 Last update 
[683dc19...3c412bc](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4018: [AIRFLOW-XXX] Log query and task_id in druid-hook

[GitBox]

codecov-io commented on issue #4018: [AIRFLOW-XXX] Log query and task_id in 
druid-hook
URL: 
https://github.com/apache/incubator-airflow/pull/4018#issuecomment-427872750
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=h1)
 Report
   > Merging 
[#4018](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/683dc19fe36afc0f653bdd1d10a08918d48be335?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4018/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4018  +/-   ##
   ==
   + Coverage   75.76%   75.77%   +<.01% 
   ==
 Files 199  199  
 Lines   1594615948   +2 
   ==
   + Hits1208212084   +2 
 Misses   3864 3864
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/hooks/druid\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4018/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kcnVpZF9ob29rLnB5)
 | `88% <100%> (+0.32%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=footer).
 Last update 
[683dc19...3c412bc](https://codecov.io/gh/apache/incubator-airflow/pull/4018?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] testvinder opened a new pull request #4019: Included Danamica in list of companies using Airflow

[GitBox]

testvinder opened a new pull request #4019: Included Danamica in list of 
companies using Airflow
URL: https://github.com/apache/incubator-airflow/pull/4019
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] awelsh93 opened a new pull request #4018: [AIRFLOW-XXX] Log query and task_id in druid-hook

[GitBox]

awelsh93 opened a new pull request #4018: [AIRFLOW-XXX] Log query and task_id 
in druid-hook
URL: https://github.com/apache/incubator-airflow/pull/4018
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   When the druid hook is used for ingestion it would be helpful to log the 
ingestion spec and the resulting task id.
   
   ### Tests
   
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   `nosetests -v tests/hooks/test_druid_hook.py` ran successfully
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-3139.

   Resolution: Fixed
Fix Version/s: 2.0.0

> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
> log INFO.
> But if parameters are used for the sql command, the parameters are not 
> printed. This makes the logging less useful.
>  
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] aoen commented on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

aoen commented on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously
URL: 
https://github.com/apache/incubator-airflow/pull/4005#issuecomment-427849285
 
 
   CI has passed, ready to be merged.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aoen commented on a change in pull request #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

aoen commented on a change in pull request #4005: [AIRFLOW-3160] Load 
latest_dagruns asynchronously
URL: https://github.com/apache/incubator-airflow/pull/4005#discussion_r223377238
 
 

 ##
 File path: airflow/www_rbac/views.py
 ##
 @@ -392,6 +392,33 @@ def task_stats(self, session=None):
 payload[dag.safe_dag_id].append(d)
 return wwwutils.json_response(payload)
 
+@expose('/last_dagruns')
+@has_access
+@provide_session
+def last_dagruns(self, session=None):
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aoen edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns asynchronously

[GitBox]

aoen edited a comment on issue #4005: [AIRFLOW-3160] Load latest_dagruns 
asynchronously
URL: 
https://github.com/apache/incubator-airflow/pull/4005#issuecomment-427398907
 
 
   Good call on the RBAC templates, will make a fix. 
   
   > Aren't we still making all the same queries?
   
   Queries are slightly different, in the past we made one query per DAG, now 
it's just a single query, so the constant overhead of each query is removed.
   
   > What's the time to the page being fully populated too please?
   
   Locally the time for the page to be fully populated is pretty much instant. 
Prod is within a second, the query is pretty much a more lightweight version of 
the one being used to fetch and display task instance state on the same page so 
performance shouldn't be a concern.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3139) Include parameters into log.info for SQL-operators if any

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641867#comment-16641867
 ] 

ASF GitHub Bot commented on AIRFLOW-3139:
-

ashb closed pull request #3986: [AIRFLOW-3139] include parameters into log.info 
in SQL operators, if any
URL: https://github.com/apache/incubator-airflow/pull/3986
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index 5b50ade34e..bb2c20c188 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -163,10 +163,11 @@ def run(self, sql, autocommit=False, parameters=None):
 for s in sql:
 if sys.version_info[0] < 3:
 s = s.encode('utf-8')
-self.log.info(s)
 if parameters is not None:
+self.log.info("{} with parameters {}".format(s, 
parameters))
 cur.execute(s, parameters)
 else:
+self.log.info(s)
 cur.execute(s)
 
 # If autocommit was set to False for db that supports autocommit,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Include parameters into log.info for SQL-operators if any
> -
>
> Key: AIRFLOW-3139
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3139
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Attachments: Screen Shot 2018-10-02 at 3.24.21 PM.png
>
>
> For SQL-operators based on {color:#a6e22e}DbApiHook{color}, like 
> SqliteOperator, PostgresOperator, the SQL command itself will be printed into 
> log INFO.
> But if parameters are used for the sql command, the parameters are not 
> printed. This makes the logging less useful.
>  
> *Example:*
> !Screen Shot 2018-10-02 at 3.24.21 PM.png|width=818,height=61!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ashb closed pull request #3986: [AIRFLOW-3139] include parameters into log.info in SQL operators, if any

[GitBox]

ashb closed pull request #3986: [AIRFLOW-3139] include parameters into log.info 
in SQL operators, if any
URL: https://github.com/apache/incubator-airflow/pull/3986
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/dbapi_hook.py b/airflow/hooks/dbapi_hook.py
index 5b50ade34e..bb2c20c188 100644
--- a/airflow/hooks/dbapi_hook.py
+++ b/airflow/hooks/dbapi_hook.py
@@ -163,10 +163,11 @@ def run(self, sql, autocommit=False, parameters=None):
 for s in sql:
 if sys.version_info[0] < 3:
 s = s.encode('utf-8')
-self.log.info(s)
 if parameters is not None:
+self.log.info("{} with parameters {}".format(s, 
parameters))
 cur.execute(s, parameters)
 else:
+self.log.info(s)
 cur.execute(s)
 
 # If autocommit was set to False for db that supports autocommit,


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #3986: [AIRFLOW-3139] include parameters into log.info in SQL operators, if any

[GitBox]

XD-DENG commented on issue #3986: [AIRFLOW-3139] include parameters into 
log.info in SQL operators, if any
URL: 
https://github.com/apache/incubator-airflow/pull/3986#issuecomment-427843280
 
 
   Hi @ashb , may you take a look? Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators

[GitBox]

wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 
operators
URL: 
https://github.com/apache/incubator-airflow/pull/3828#issuecomment-427842299
 
 
   The new updates have been pushed and verified. Please review. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators

[GitBox]

codecov-io edited a comment on issue #3828: [AIRFLOW-2993] s3_to_sftp and 
sftp_to_s3 operators
URL: 
https://github.com/apache/incubator-airflow/pull/3828#issuecomment-417764201
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=h1)
 Report
   > Merging 
[#3828](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/ac87e15201d5a662639a2c79bb74149fe138dc70?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/3828/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#3828  +/-   ##
   ==
   - Coverage   75.79%   75.77%   -0.02% 
   ==
 Files 199  199  
 Lines   1594515945  
   ==
   - Hits1208512082   -3 
   - Misses   3860 3863   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3828/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `82.13% <0%> (-0.27%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=footer).
 Last update 
[ac87e15...c4015ee](https://codecov.io/gh/apache/incubator-airflow/pull/3828?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP

[GitBox]

ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server 
certificate when connecting to LDAP
URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r223367174
 
 

 ##
 File path: airflow/contrib/auth/backends/ldap_auth.py
 ##
 @@ -55,16 +55,20 @@ class LdapException(Exception):
 
 
 def get_ldap_connection(dn=None, password=None):
-tls_configuration = None
-use_ssl = False
+cacert = None
 try:
 cacert = configuration.conf.get("ldap", "cacert")
-tls_configuration = Tls(validate=ssl.CERT_REQUIRED, 
ca_certs_file=cacert)
-use_ssl = True
-except Exception:
+except AirflowConfigException:
 pass
 
-server = Server(configuration.conf.get("ldap", "uri"), use_ssl, 
tls_configuration)
+tls_configuration = Tls(validate=ssl.CERT_REQUIRED,
+version=ssl.PROTOCOL_SSLv23,
 
 Review comment:
   Do we really want to say SSL v2? Isn't that a security risk? Default to 
TLSv1.2 with a config option to switch back?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server certificate when connecting to LDAP

[GitBox]

ashb commented on a change in pull request #4006: [AIRFLOW-3164] Verify server 
certificate when connecting to LDAP
URL: https://github.com/apache/incubator-airflow/pull/4006#discussion_r223367394
 
 

 ##
 File path: docs/security.rst
 ##
 @@ -60,7 +60,7 @@ LDAP
 
 To turn on LDAP authentication configure your ``airflow.cfg`` as follows. 
Please note that the example uses
 an encrypted connection to the ldap server as you probably do not want 
passwords be readable on the network level.
-It is however possible to configure without encryption if you really want to.
+We do not support insecure connections anymore.
 
 Review comment:
   Note in UPDATING.md please.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table

[Aaron Dossett (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641820#comment-16641820
 ] 

Aaron Dossett commented on AIRFLOW-2516:


[~xnuinside] Are you saying this is fixed in 1.10?

> Deadlock found when trying to update task_instance table
> 
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
>
>  
>  
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask: 
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue: 
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227 
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback 
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/bin/airflow", line 27, in 
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: 
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in 
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result = 
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in 
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in 
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: 
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: 
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged = 
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882, 
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident, 
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952, 
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247, 
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return 
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884, 
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854, 
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret = 
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925, 
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return 
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File 
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946, 
> in _execute_and_instances
> [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: 
> close_with_result=True)
> [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask: File 
> 

[jira] [Resolved] (AIRFLOW-2717) FileToGoogleCloudStorageOperator not shown in the Documentation.

[Ash Berlin-Taylor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ash Berlin-Taylor resolved AIRFLOW-2717.

   Resolution: Fixed
Fix Version/s: (was: 1.9.0)

> FileToGoogleCloudStorageOperator not shown in the Documentation.
> 
>
> Key: AIRFLOW-2717
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2717
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib, Documentation, gcp, operators
>Affects Versions: 1.9.0
>Reporter: Michele De Simoni
>Priority: Minor
>  Labels: docuentation, easyfix
>
> [FileToGoogleCloudStorageOperator|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/file_to_gcs.py]
>  is present in the codebase but not in the 
> [documentation|https://airflow.incubator.apache.org/code.html#community-contributed-operators].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] msumit commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for airflow added

[GitBox]

msumit commented on issue #3989: [AIRFLOW-1945] Autoscale celery workers for 
airflow added
URL: 
https://github.com/apache/incubator-airflow/pull/3989#issuecomment-427814003
 
 
   @phani8996 whats harm in letting my machine run with the full capacity of 
workers all the time? Cause if I am allowing to grow it to a max, then it means 
that my machine has the capacity to handle that many workers anyway. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] msumit commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of logs instead of rendering whole log

[GitBox]

msumit commented on issue #3992: [AIRFLOW-620] Feature to tail custom number of 
logs instead of rendering whole log
URL: 
https://github.com/apache/incubator-airflow/pull/3992#issuecomment-427812670
 
 
   @phani8996 can you post some gifs on this feature in work?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] msumit closed pull request #4016: [AIRFLOW-XXX] Fix Typo in SFTPOperator docstring

[GitBox]

msumit closed pull request #4016: [AIRFLOW-XXX] Fix Typo in SFTPOperator 
docstring
URL: https://github.com/apache/incubator-airflow/pull/4016
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/contrib/operators/sftp_operator.py 
b/airflow/contrib/operators/sftp_operator.py
index a3b5c1f244..620d875f89 100644
--- a/airflow/contrib/operators/sftp_operator.py
+++ b/airflow/contrib/operators/sftp_operator.py
@@ -30,7 +30,7 @@ class SFTPOperation(object):
 class SFTPOperator(BaseOperator):
 """
 SFTPOperator for transferring files from remote host to local or vice a 
versa.
-This operator uses ssh_hook to open sftp trasport channel that serve as 
basis
+This operator uses ssh_hook to open sftp transport channel that serve as 
basis
 for file transfer.
 
 :param ssh_hook: predefined ssh_hook to use for remote execution.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] msumit closed pull request #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} description

[GitBox]

msumit closed pull request #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} 
description
URL: https://github.com/apache/incubator-airflow/pull/4017
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docs/code.rst b/docs/code.rst
index e5a53193e6..764d6949ac 100644
--- a/docs/code.rst
+++ b/docs/code.rst
@@ -251,7 +251,7 @@ VariableDescription
 ``{{ prev_ds_nodash }}``the previous execution date as 
``MMDD`` if exists, else ``None`
 ``{{ next_ds }}``   the next execution date as ``-MM-DD``
 if ``{{ ds }}`` is ``2016-01-01`` and 
``schedule_interval`` is ``@weekly``,
-``{{ prev_ds }}`` will be ``2016-01-08``
+``{{ next_ds }}`` will be ``2016-01-08``
 ``{{ next_ds_nodash }}``the next execution date as ``MMDD`` if 
exists, else ``None`
 ``{{ yesterday_ds }}``  yesterday's date as ``-MM-DD``
 ``{{ yesterday_ds_nodash }}``   yesterday's date as ``MMDD``


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] msumit commented on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} description

[GitBox]

msumit commented on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} 
description
URL: 
https://github.com/apache/incubator-airflow/pull/4017#issuecomment-427811086
 
 
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} description

[GitBox]

codecov-io edited a comment on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds 
}} description
URL: 
https://github.com/apache/incubator-airflow/pull/4017#issuecomment-42766
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=h1)
 Report
   > Merging 
[#4017](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/3fb977603184054b494ab1f796bf8eeefe20957d?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4017/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#4017   +/-   ##
   ===
 Coverage   75.77%   75.77%   
   ===
 Files 199  199   
 Lines   1594515945   
   ===
 Hits1208212082   
 Misses   3863 3863
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=footer).
 Last update 
[3fb9776...44ab783](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} description

[GitBox]

codecov-io commented on issue #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} 
description
URL: 
https://github.com/apache/incubator-airflow/pull/4017#issuecomment-42766
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=h1)
 Report
   > Merging 
[#4017](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/3fb977603184054b494ab1f796bf8eeefe20957d?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4017/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#4017   +/-   ##
   ===
 Coverage   75.77%   75.77%   
   ===
 Files 199  199   
 Lines   1594515945   
   ===
 Hits1208212082   
 Misses   3863 3863
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=footer).
 Last update 
[3fb9776...44ab783](https://codecov.io/gh/apache/incubator-airflow/pull/4017?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gigkokman opened a new pull request #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds }} description

[GitBox]

gigkokman opened a new pull request #4017: [AIRFLOW-XXX] Fix wrong {{ next_ds 
}} description
URL: https://github.com/apache/incubator-airflow/pull/4017
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ x] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [x ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [x ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [x ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
   
   ### Code Quality
   
   - [x ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gerardo commented on issue #3815: [AIRFLOW-2973] Add Python 3.6 to Supported Prog Langs

[GitBox]

gerardo commented on issue #3815: [AIRFLOW-2973] Add Python 3.6 to Supported 
Prog Langs
URL: 
https://github.com/apache/incubator-airflow/pull/3815#issuecomment-427762543
 
 
   > @kaxil sure! I have time this week to work on this.
   
   BTW, I hugely underestimated the amount of free time I had available 
   
   Here's the PR that creates separate images for python2 and python3: 
https://github.com/apache/incubator-airflow-ci/pull/4
   
   @tedmiston Regarding python3.7, it's still not available to any Ubuntu 
version, not even through a PPA. Even [cosmic 
cuttlefish](https://wiki.ubuntu.com/CosmicCuttlefish), which hasn't been 
released yet, supports only [3.6](https://packages.ubuntu.com/cosmic/python3).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-824) Allow writing to XCOM values via API

[Robin Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641534#comment-16641534
 ] 

Robin Miller commented on AIRFLOW-824:
--

This was to facilitate any preceding systems (e.g. one that creates a DAG Run 
via the API) to communicate into that run as well. We did not end up needing to 
use this functionality as it was more convenient to use the conf data that can 
be passed in when creating the DAG Run.

> Allow writing to XCOM values via API
> 
>
> Key: AIRFLOW-824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-824
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Robin Miller
>Assignee: Robin Miller
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] gerardo commented on issue #4: Create separate images with python2 and python3 support

[GitBox]

gerardo commented on issue #4: Create separate images with python2 and python3 
support
URL: 
https://github.com/apache/incubator-airflow-ci/pull/4#issuecomment-427760478
 
 
   @Fokko @tedmiston @kaxil please have a look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] gerardo opened a new pull request #4: Create separate images with python2 and python3 support

[GitBox]

gerardo opened a new pull request #4: Create separate images with python2 and 
python3 support
URL: https://github.com/apache/incubator-airflow-ci/pull/4
 
 
   This is required by https://github.com/apache/incubator-airflow/pull/3815 
and https://github.com/apache/incubator-airflow/pull/3816.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3170) Jinja templating error - to be clearer

[jack (JIRA)]

jack created AIRFLOW-3170:
-

 Summary: Jinja templating error - to be clearer 
 Key: AIRFLOW-3170
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3170
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.10.0
Reporter: jack


Consider the following code:
{code:java}
   import_orders_op = MySqlToGoogleCloudStorageOperator(
    task_id='import_orders',
    mysql_conn_id='mysql_conn',
    google_cloud_storage_conn_id='gcp_con',
    provide_context=True,
    sql=''' SELECT * FROM {{ params.table_name }} WHERE orders_id > {{ 
params.last_imported_id }} AND orders_id < {{ ti.xcom_pull('get_max_order_id') 
}}''',
    params={'last_imported_id': LAST_IMPORTED_ORDER_ID, 'table_name' :  
BQ_TABLE_NAME},
    bucket=GCS_BUCKET_ID,
    filename=file_name,
    dag=dag){code}
 

It gives me:
{code:java}
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 1:
 invalid start byte{code}
 

 

This error is unclear. It doesn't say where the problem is or what is the 
actual cause. It took me hours to trace it to the *missing space* between
{code:java}
{{ ti.xcom_pull('get_max_order_id') }}{code}
and the
{code:java}
 '''{code}
 

 

Please improve the error message to explain the +actual problem+. A clear error 
message that refer to the syntax error near the  '''  would help a lot. This is 
the kind of error where programmers may lose a lot of time to find because it's 
very unnoticeable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] cpnat commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism

[GitBox]

cpnat commented on issue #3584: [AIRFLOW-249] Refactor the SLA mechanism
URL: 
https://github.com/apache/incubator-airflow/pull/3584#issuecomment-427753607
 
 
   Hello @Eronarn,
   
   Do you plan to continue working on this PR? SLA on task duration is 
definitely a useful feature (for myself at least!) :-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3145) [Edit Docummntation ] reduce airflow dag scheduling latency in production

[jack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jack updated AIRFLOW-3145:
--
Component/s: docs
Summary: [Edit Docummntation ] reduce airflow dag scheduling latency in 
production  (was: reduce airflow dag scheduling latency in production)

> [Edit Docummntation ] reduce airflow dag scheduling latency in production
> -
>
> Key: AIRFLOW-3145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3145
> Project: Apache Airflow
>  Issue Type: Task
>  Components: docs
>Affects Versions: 1.10.1
>Reporter: jack
>Priority: Trivial
>
> The docs says:
>  * {{}}
> {code:java}
> max_threads: Scheduler will spawn multiple threads in parallel to schedule 
> dags. This is controlled by max_threads with default value of 2. User should 
> increase this value to a larger value(e.g numbers of cpus where scheduler 
> runs - 1) in production.{code}
> [https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
>  
>  
> However in the Aiflow.cfg the comment is
> {code:java}
> #The scheduler can run multiple threads in parallel to schedule dags.
> # This defines how many threads will run.
> max_threads = 2{code}
>  
>  
> May I suggest that the
> {code:java}
> User should increase this value to a larger value(e.g numbers of cpus where 
> scheduler runs - 1) in production.{code}
>  
> will also be written in the comment at Aiflow.cfg 
> Not everyone read the docs from top to bottom this could be easily missed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ron819 commented on issue #4003: [AIRFLOW-3163] add operator to enable setting table description in BigQuery table

[GitBox]

ron819 commented on issue #4003: [AIRFLOW-3163] add operator to enable setting 
table description in BigQuery table
URL: 
https://github.com/apache/incubator-airflow/pull/4003#issuecomment-427749891
 
 
   @ant I believe that most people add their `etl_timestamp` as a column to 
the table itself. In some scenario it's even the partition field of the table.
   
   I think the better approach is to ask Google to add support for Description 
& Column description in their CLI commands. Currently it's supported only 
manually from the UI. 
   https://cloud.google.com/support/docs/issue-trackers
   
   Once google will add this feature it will be possible to modify the Airflow 
Operator to support it.
   
   I don't think there should be a separated operator just for this option. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vardancse commented on a change in pull request #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition

[GitBox]

vardancse commented on a change in pull request #3994: [AIRFLOW-3136] Add 
retry_number to TaskInstance Key property to avoid race condition
URL: https://github.com/apache/incubator-airflow/pull/3994#discussion_r223258140
 
 

 ##
 File path: airflow/models.py
 ##
 @@ -1230,7 +1230,7 @@ def key(self):
 """
 Returns a tuple that identifies the task instance uniquely
 """
-return self.dag_id, self.task_id, self.execution_date
+return self.dag_id, self.task_id, self.execution_date, self.try_number
 
 Review comment:
   @ashb Looking forward to your feedback!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services