[jira] [Updated] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-17 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector updated AIRFLOW-5176:
-
Description: 
Add a hook and an operator that allow sending queries to Azure Data Explorer 
(Kusto) cluster.

ADX (Azure Data Explorer) is relatively new but very promising analytics data 
store / data processing offering in Azure:

[https://azure.microsoft.com/en-us/services/data-explorer/]

 

PR: [https://github.com/apache/airflow/pull/5785]

  was:
Add a hook and an operator that allow sending queries to Azure Data Explorer 
(Kusto) cluster.

ADX (Azure Data Explorer) is relatively new but very promising analytics data 
store / data processing offering in Azure.

PR: https://github.com/apache/airflow/pull/5785


> Add integration with Azure Data Explorer
> 
>
> Key: AIRFLOW-5176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.4, 2.0.0
>Reporter: Michael Spector
>Assignee: Michael Spector
>Priority: Major
>
> Add a hook and an operator that allow sending queries to Azure Data Explorer 
> (Kusto) cluster.
> ADX (Azure Data Explorer) is relatively new but very promising analytics data 
> store / data processing offering in Azure:
> [https://azure.microsoft.com/en-us/services/data-explorer/]
>  
> PR: [https://github.com/apache/airflow/pull/5785]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-17 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector reassigned AIRFLOW-5176:


Assignee: Michael Spector

> Add integration with Azure Data Explorer
> 
>
> Key: AIRFLOW-5176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.4, 2.0.0
>Reporter: Michael Spector
>Assignee: Michael Spector
>Priority: Major
>
> Add a hook and an operator that allow sending queries to Azure Data Explorer 
> (Kusto) cluster.
> ADX (Azure Data Explorer) is relatively new but very promising analytics data 
> store / data processing offering in Azure.
> PR: https://github.com/apache/airflow/pull/5785



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Assigned] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-14 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector reassigned AIRFLOW-5176:


Assignee: (was: Michael Spector)

> Add integration with Azure Data Explorer
> 
>
> Key: AIRFLOW-5176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.4, 2.0.0
>Reporter: Michael Spector
>Priority: Major
>
> Add a hook and an operator that allow sending queries to Azure Data Explorer 
> (Kusto) cluster.
> ADX (Azure Data Explorer) is relatively new but very promising analytics data 
> store / data processing offering in Azure.
> PR: https://github.com/apache/airflow/pull/5785



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-13 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector updated AIRFLOW-5176:
-
Description: 
Add a hook and an operator that allow sending queries to Azure Data Explorer 
(Kusto) cluster.

ADX (Azure Data Explorer) is relatively new but very promising analytics data 
store / data processing offering in Azure.

PR: https://github.com/apache/airflow/pull/5785

  was:
Add a hook and an operator that allow sending queries to Azure Data Explorer 
(Kusto) cluster.

ADX is relatively new but very promising analytics data store / data processing 
offering in Azure.


> Add integration with Azure Data Explorer
> 
>
> Key: AIRFLOW-5176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.4, 2.0.0
>Reporter: Michael Spector
>Assignee: Michael Spector
>Priority: Major
>
> Add a hook and an operator that allow sending queries to Azure Data Explorer 
> (Kusto) cluster.
> ADX (Azure Data Explorer) is relatively new but very promising analytics data 
> store / data processing offering in Azure.
> PR: https://github.com/apache/airflow/pull/5785



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-12 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector updated AIRFLOW-5176:
-
Affects Version/s: 1.10.4

> Add integration with Azure Data Explorer
> 
>
> Key: AIRFLOW-5176
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: hooks, operators
>Affects Versions: 1.10.4, 2.0.0
>Reporter: Michael Spector
>Assignee: Michael Spector
>Priority: Major
>
> Add a hook and an operator that allow sending queries to Azure Data Explorer 
> (Kusto) cluster.
> ADX is relatively new but very promising analytics data store / data 
> processing offering in Azure.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (AIRFLOW-5176) Add integration with Azure Data Explorer

2019-08-12 Thread Michael Spector (JIRA) 
Michael Spector created AIRFLOW-5176:


 Summary: Add integration with Azure Data Explorer
 Key: AIRFLOW-5176
 URL: https://issues.apache.org/jira/browse/AIRFLOW-5176
 Project: Apache Airflow
  Issue Type: New Feature
  Components: hooks, operators
Affects Versions: 2.0.0
Reporter: Michael Spector
Assignee: Michael Spector


Add a hook and an operator that allow sending queries to Azure Data Explorer 
(Kusto) cluster.

ADX is relatively new but very promising analytics data store / data processing 
offering in Azure.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Closed] (AIRFLOW-3567) Logs are not fetched from S3 when using AIM policy

2018-12-26 Thread Michael Spector (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Spector closed AIRFLOW-3567.

Resolution: Invalid

There was missing rule in the AIM policy.

> Logs are not fetched from S3 when using AIM policy
> --
>
> Key: AIRFLOW-3567
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3567
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: logging
>Affects Versions: 1.10.1
>Reporter: Michael Spector
>Priority: Major
>
> The remote logging is configured as follows:
> {code:python}
> remote_logging=True
> remote_log_conn_id=''
> remote_base_log_folder='s3://logs-bucket/airflow/analytics-airflow'
> encrypt_s3_logs=False
> {code}
> Logs on S3 are created successfully, however when trying to access them via 
> UI we're getting something like this:
> {code:bash}
> Log file does not exist: 
> /var/lib/airflow/logs/jobname/taskname/2018-12-25T03:00:00+00:00/1.log 
> Fetching from: 
> http://10.118.19.142:50192/log/jobname/taskname/2018-12-25T03:00:00+00:00/1.log
> {code}
> Which means that logs fetcher falls back to FileTaskHandler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3567) Logs are not fetched from S3 when using AIM policy

2018-12-25 Thread Michael Spector (JIRA) 
Michael Spector created AIRFLOW-3567:


 Summary: Logs are not fetched from S3 when using AIM policy
 Key: AIRFLOW-3567
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3567
 Project: Apache Airflow
  Issue Type: Bug
  Components: logging
Affects Versions: 1.10.1
Reporter: Michael Spector


The remote logging is configured as follows:

{code:python}
remote_logging=True
remote_log_conn_id=''
remote_base_log_folder='s3://logs-bucket/airflow/analytics-airflow'
encrypt_s3_logs=False
{code}

Logs on S3 are created successfully, however when trying to access them via UI 
we're getting something like this:

{code:bash}
Log file does not exist: 
/var/lib/airflow/logs/jobname/taskname/2018-12-25T03:00:00+00:00/1.log 
Fetching from: 
http://10.118.19.142:50192/log/jobname/taskname/2018-12-25T03:00:00+00:00/1.log
{code}

Which means that logs fetcher falls back to FileTaskHandler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)