Re: Good jinja templating in airflow examples

2017-01-18 Thread Maxime Beauchemin
Here's the list of variables and macros exposed in the jinja context: https://airflow.apache.org/code.html#macros Those are all exposed by the framework, meaning they are usable for any templated field without doing anything special. There are ways to pass your own variables and methods using the

Re: Question - HttpHook

2017-01-18 Thread Hanneli Tavante
Thank you very much for the reply! I set these parameters using the web interface. I tried several combos for an HTTP type: - setting host, login, password only - setting schema field with https + setting host, login, password only - setting port 443 + setting host, login, password only - setting p

Re: Question - HttpHook

2017-01-18 Thread Matt Hickman
Couple things: - You shouldn’t need to set port 443 - Make sure the url starts with https:// in the connection settings - To check your auth is working properly try something like `print api_hook.get_conn(None).auth` - My connection in airflow has a trailing / and when I run your code below

Re: Question - HttpHook

2017-01-18 Thread Matt Hickman
Hanneli, We hit the data bricks api from Airflow. The connection info for our data bricks instance is pretty simple: conn type: http host: https://.cloud.databricks.com/ login: password: Knowing what error you get would be helpful! Also if you provide the code your using for your hook. Cheers

Question - HttpHook

2017-01-18 Thread Hanneli Tavante
Hi! Could anybody help me? I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8; Databricks API 2.0 and Python 2.7.6 ) Is there any example with that? It looks like I can't authenticate. I created an HTTP connection on 'Connections', inserted my credentials and pointed to the po

Re: Airflow 1.8.0 BETA 1

2017-01-18 Thread Chris Riccomini
We are switching to 1.8.0b1 this week--both dev and prod. Will keep you posted. On Wed, Jan 18, 2017 at 11:51 AM, Alex Van Boxel wrote: > Hey Max, > > As I'm missing the 1.7.2 labels I compared to the 172 branch. Can you have > a look at PR 2000. Its also sanitised, removing some of the commits

Re: Airflow 1.8.0 BETA 1

2017-01-18 Thread Alex Van Boxel
Hey Max, As I'm missing the 1.7.2 labels I compared to the 172 branch. Can you have a look at PR 2000. Its also sanitised, removing some of the commits that doesn't bring value to the users. On Wed, Jan 18, 2017, 02:51 Maxime Beauchemin wrote: > Alex, for the CHANGELOG.md, I've been using `gith

Re: Airflow 1.8.0 BETA 1

2017-01-18 Thread Bolke de Bruin
Just to let you know: we just switched to Airflow 1.8.0b1 in production. Fingers crossed :) Bolke > On 18 Jan 2017, at 02:50, Maxime Beauchemin > wrote: > > Alex, for the CHANGELOG.md, I've been using `github-changes`, a js app that > make changelog generation flexible and easy. > > https://

Re: Good jinja templating in airflow examples

2017-01-18 Thread Boris Tyukin
Hi Guilherme, I guess it depends what exactly you want to do as not everything works with jinja. >From documentation: https://pythonhosted.org/airflow/concepts.html#jinja-templating You can use Jinja templating with every parameter that is marked as “templated” in the documentation. You can op

Good jinja templating in airflow examples

2017-01-18 Thread Guilherme Marthe
Hey folks! I am trying to write a dag that works well with jinja templating, since through my study of the documentation, is the best way to ensure compatibility with the back-fill function. Are you guys aware of any examples online with this functionality working? The, documentation is still a b