Re: Good jinja templating in airflow examples

2017-01-18 Thread Maxime Beauchemin 
Here's the list of variables and macros exposed in the jinja context:
https://airflow.apache.org/code.html#macros

Those are all exposed by the framework, meaning they are usable for any
templated field without doing anything special.

There are ways to pass your own variables and methods using the `params`
attribute of any operator as shown in the tutorial here
, and
there's also a `user_defined_macros` parameter as a dict you can pass while
creating the DAG object. Every key in that dictionary is made available in
the global jinja namespace.

https://airflow.apache.org/code.html#airflow.models.DAG

Max

On Wed, Jan 18, 2017 at 11:31 AM, Boris Tyukin 
wrote:

> Hi Guilherme,
>
> I guess it depends what exactly you want to do as not everything works with
> jinja.
>
> From documentation:
>
> https://pythonhosted.org/airflow/concepts.html#jinja-templating
>
> You can use Jinja templating with every parameter that is marked as
> “templated” in the documentation.
>
> You can open source code for operators and see what parameters are actually
> templated.
>
> For example, if you open source code for BashOperator
> https://pythonhosted.org/airflow/_modules/bash_operator.html#BashOperator
> you will see this line of code:
> template_fields = ('bash_command', 'env')
>
> It means only bash_command and env will work with jinja templates.
>
> On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
> guilherme.mar...@enjoei.com.br> wrote:
>
> > Hey folks!
> >
> > I am trying to write a dag that works well with jinja templating, since
> > through my study of the documentation, is the best way to ensure
> > compatibility with the back-fill function.
> >
> > Are you guys aware of any examples online with this functionality
> working?
> > The, documentation is still a bit dry on insightful examples, and I am a
> > noobie developer, so I am trying to make the best use of the tool in the
> > "correct way".
> >
> > Thank you in advance for any resources you guys can share!
> >
> > Gui
> >
> > Ps: I ve been told I can email you guys with questions like these :) Hope
> > this is not a hassle ^_^
> >
>

Re: Question - HttpHook

2017-01-18 Thread Matt Hickman 
Couple things:
  - You shouldn’t need to set port 443
  - Make sure the url starts with https:// in the connection settings
  - To check your auth is working properly try something like `print
api_hook.get_conn(None).auth`
  - My connection in airflow has a trailing / and when I run your code
below with the endpoint beginning with a “/“ I also get some html in the
response. Try changing `api_hook.run('/api/2.0/jobs/list’)` to
`api_hook.run('api/2.0/jobs/list’)`.

Cheers,
Matt

On January 18, 2017 at 4:48:19 PM, Hanneli Tavante (hannel...@gmail.com)
wrote:

Thank you very much for the reply!
I set these parameters using the web interface. I tried several combos for
an HTTP type:
- setting host, login, password only
- setting schema field with https + setting host, login, password only
- setting port 443 + setting host, login, password only
- setting port 443 + schema field https + setting host, login, password
only

for a certain my_dag.py, I run a PythonOperator calling a method that does
the following:
``` api_hook =
HttpHook(http_conn_id='databricks_dcs_convert_json_to_parquet_raw_workflow_job_id',
method='GET')

response = api_hook.run('/api/2.0/jobs/list')
```

I also tried a complete version of run method:
```
response = api_hook.run('/api/2.0/jobs/list', None, 'application/json',
None)

```
The problem is the same - I end up on the Login page.

(I know my credentials are fine since I can perform a plain curl)

It is odd since I succeed for other API integrations.

Thank you very much for the help!


On Wed, Jan 18, 2017 at 4:38 PM Matt Hickman  wrote:

Hanneli,

We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:

conn type: http
host: https://.cloud.databricks.com/
login: 
password: 

Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.

Cheers,
Matt

On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannel...@gmail.com)
wrote:

Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!

Re: Question - HttpHook

2017-01-18 Thread Matt Hickman 
Hanneli,

We hit the data bricks api from Airflow. The connection info for our data
bricks instance is pretty simple:

conn type: http
host: https://.cloud.databricks.com/
login: 
password: 

Knowing what error you get would be helpful! Also if you provide the code
your using for your hook.

Cheers,
Matt

On January 18, 2017 at 4:33:07 PM, Hanneli Tavante (hannel...@gmail.com)
wrote:

Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!

Question - HttpHook

2017-01-18 Thread Hanneli Tavante 
Hi! Could anybody help me?
I am trying to use the HTTPHook to connect with Databricks. (Airflow 1.8;
Databricks API 2.0 and Python 2.7.6 )
Is there any example with that? It looks like I can't authenticate. I
created an HTTP connection on 'Connections', inserted my credentials and
pointed to the port 443, but I did not succeed.
Thanks!

Re: Airflow 1.8.0 BETA 1

2017-01-18 Thread Chris Riccomini 
We are switching to 1.8.0b1 this week--both dev and prod. Will keep you
posted.

On Wed, Jan 18, 2017 at 11:51 AM, Alex Van Boxel  wrote:

> Hey Max,
>
> As I'm missing the 1.7.2 labels I compared to the 172 branch. Can you have
> a look at PR 2000. Its also sanitised, removing some of the commits that
> doesn't bring value to the users.
>
> On Wed, Jan 18, 2017, 02:51 Maxime Beauchemin 
> wrote:
>
> > Alex, for the CHANGELOG.md, I've been using `github-changes`, a js app
> that
> > make changelog generation flexible and easy.
> >
> > https://www.npmjs.com/package/github-changes
> >
> > Command looks something like:
> > `github-changes -o apache -r incubator-airflow --token  TOKEN>
> > --between-tags 1.7.2...1.8.0beta` (tags may differ, it's easy to get a
> > token on your GH profile page)
> >
> > This will write a `CHANGELOG.md` in your cwd that you can just add on top
> > of the existing one
> >
> > Max
> >
> > On Jan 17, 2017 3:37 PM, "Dan Davydov" 
> > wrote:
> >
> > > So it is, my bad. Bad skills with ctrl-f :).
> > >
> > > On Tue, Jan 17, 2017 at 3:31 PM, Bolke de Bruin 
> > wrote:
> > >
> > > > Arthur's change is already in!
> > > >
> > > > B.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 17 Jan 2017, at 22:20, Dan Davydov  > .INVALID>
> > > > wrote:
> > > > >
> > > > > Would be good to cherrypick Arthur's fix into here if possible:
> > > > > https://github.com/apache/incubator-airflow/pull/1973/files
> (commit
> > > > > 43bf89d)
> > > > >
> > > > > The impersonation stuff should be wrapping up shortly pending
> Bolke's
> > > > > comments.
> > > > >
> > > > > Also agreed with Max on the thanks. Thanks Alex too for the change
> > log!
> > > > >
> > > > > On Tue, Jan 17, 2017 at 10:05 AM, Maxime Beauchemin <
> > > > > maximebeauche...@gmail.com> wrote:
> > > > >
> > > > >> Bolke, I couldn't thank you enough for driving the release
> process!
> > > > >>
> > > > >> I'll coordinate with the Airbnb team around impersonation/CGROUPs
> > and
> > > on
> > > > >> making sure we put this release in our staging ASAP. We have our
> > > > employee
> > > > >> conference this week so things are slower, but we'll be back at
> full
> > > > speed
> > > > >> Friday.
> > > > >>
> > > > >> Max
> > > > >>
> > > > >>> On Mon, Jan 16, 2017 at 3:51 PM, Alex Van Boxel <
> a...@vanboxel.be>
> > > > wrote:
> > > > >>>
> > > > >>> Hey Bolke, thanks great wotk. I'll handle the CHANGELOG, and add
> > some
> > > > >>> documentation about triggers with branching operators.
> > > > >>>
> > > > >>> About the Google Cloud Operators: I wouldn't call it feature
> > > > complete...
> > > > >> it
> > > > >>> never is.
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jan 16, 2017 at 11:24 PM Bolke de Bruin <
> bdbr...@gmail.com
> > >
> > > > >> wrote:
> > > > >>>
> > > >  Dear All,
> > > > 
> > > >  I have made the first BETA of Airflow 1.8.0 available at:
> > > >  https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
> > > >  https://dist.apache.org/repos/dist/dev/incubator/airflow/> ,
> > public
> > > > >> keys
> > > >  are available at
> > > >  https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> > > >  https://dist.apache.org/repos/dist/release/incubator/airflow/>
> .
> > It
> > > > is
> > > >  tagged with a local version “apache.incubating” so it allows
> > > upgrading
> > > > >>> from
> > > >  earlier releases. This beta is available for testing in a more
> > > > >> production
> > > >  like setting (acceptance environment?).
> > > > 
> > > >  I would like to encourage everyone  to try it out, to report
> back
> > > any
> > > >  issues so we get to a rock solid release of 1.8.0. When
> reporting
> > > > >> issues
> > > > >>> a
> > > >  test case or even a fix is highly appreciated.
> > > > 
> > > >  By moving to beta, we are also in feature freeze mode. Meaning
> no
> > > > major
> > > >  adjustments or additions can be made to the v1-8-test branch.
> > There
> > > is
> > > > >>> one
> > > >  exception: the cgroups+impersonation patch. I was assured that
> > > before
> > > > >> we
> > > >  merge that it will be thoroughly tested, so its can still enter
> > 1.8
> > > if
> > > >  within a reasonable time frame. A lot of work has gone into it
> and
> > > it
> > > > >>> would
> > > >  be a shame if we would lose momentum.
> > > > 
> > > >  Finally, it would also be really nice of have some updates to
> the
> > > >  documentation. In order of importance:
> > > > 
> > > >  * UPDATING.md What does a user need to think of when upgrading
> to
> > > 1.8?
> > > >  MySQL 5.6.4 is now minimally required, scheduler now has
> separate
> > > logs
> > > > >>> per
> > > >  file processor.
> > > >  * docs/configuration.rst We have many new options, especially in
> > the
> > > >

Re: Airflow 1.8.0 BETA 1

2017-01-18 Thread Bolke de Bruin 
Just to let you know: we just switched to Airflow 1.8.0b1 in production. 
Fingers crossed :)

Bolke

> On 18 Jan 2017, at 02:50, Maxime Beauchemin  
> wrote:
> 
> Alex, for the CHANGELOG.md, I've been using `github-changes`, a js app that
> make changelog generation flexible and easy.
> 
> https://www.npmjs.com/package/github-changes
> 
> Command looks something like:
> `github-changes -o apache -r incubator-airflow --token 
> --between-tags 1.7.2...1.8.0beta` (tags may differ, it's easy to get a
> token on your GH profile page)
> 
> This will write a `CHANGELOG.md` in your cwd that you can just add on top
> of the existing one
> 
> Max
> 
> On Jan 17, 2017 3:37 PM, "Dan Davydov" 
> wrote:
> 
>> So it is, my bad. Bad skills with ctrl-f :).
>> 
>> On Tue, Jan 17, 2017 at 3:31 PM, Bolke de Bruin  wrote:
>> 
>>> Arthur's change is already in!
>>> 
>>> B.
>>> 
>>> Sent from my iPhone
>>> 
 On 17 Jan 2017, at 22:20, Dan Davydov 
>>> wrote:
 
 Would be good to cherrypick Arthur's fix into here if possible:
 https://github.com/apache/incubator-airflow/pull/1973/files (commit
 43bf89d)
 
 The impersonation stuff should be wrapping up shortly pending Bolke's
 comments.
 
 Also agreed with Max on the thanks. Thanks Alex too for the change log!
 
 On Tue, Jan 17, 2017 at 10:05 AM, Maxime Beauchemin <
 maximebeauche...@gmail.com> wrote:
 
> Bolke, I couldn't thank you enough for driving the release process!
> 
> I'll coordinate with the Airbnb team around impersonation/CGROUPs and
>> on
> making sure we put this release in our staging ASAP. We have our
>>> employee
> conference this week so things are slower, but we'll be back at full
>>> speed
> Friday.
> 
> Max
> 
>> On Mon, Jan 16, 2017 at 3:51 PM, Alex Van Boxel 
>>> wrote:
>> 
>> Hey Bolke, thanks great wotk. I'll handle the CHANGELOG, and add some
>> documentation about triggers with branching operators.
>> 
>> About the Google Cloud Operators: I wouldn't call it feature
>>> complete...
> it
>> never is.
>> 
>> 
>> On Mon, Jan 16, 2017 at 11:24 PM Bolke de Bruin 
> wrote:
>> 
>>> Dear All,
>>> 
>>> I have made the first BETA of Airflow 1.8.0 available at:
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public
> keys
>>> are available at
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It
>>> is
>>> tagged with a local version “apache.incubating” so it allows
>> upgrading
>> from
>>> earlier releases. This beta is available for testing in a more
> production
>>> like setting (acceptance environment?).
>>> 
>>> I would like to encourage everyone  to try it out, to report back
>> any
>>> issues so we get to a rock solid release of 1.8.0. When reporting
> issues
>> a
>>> test case or even a fix is highly appreciated.
>>> 
>>> By moving to beta, we are also in feature freeze mode. Meaning no
>>> major
>>> adjustments or additions can be made to the v1-8-test branch. There
>> is
>> one
>>> exception: the cgroups+impersonation patch. I was assured that
>> before
> we
>>> merge that it will be thoroughly tested, so its can still enter 1.8
>> if
>>> within a reasonable time frame. A lot of work has gone into it and
>> it
>> would
>>> be a shame if we would lose momentum.
>>> 
>>> Finally, it would also be really nice of have some updates to the
>>> documentation. In order of importance:
>>> 
>>> * UPDATING.md What does a user need to think of when upgrading to
>> 1.8?
>>> MySQL 5.6.4 is now minimally required, scheduler now has separate
>> logs
>> per
>>> file processor.
>>> * docs/configuration.rst We have many new options, especially in the
>>> scheduler area
>>> * docs/faq.rst
>>> * CHANGELOG.txt (compiled from git log)
>>> * swagger definitions for the API
>>> 
>>> HIGHLIGHTS of the beta:
>>> 
>>> * DAG catchup: If False the scheduler does not fill in the gaps
>>> between
>>> the start_date and the current_date. Can be specified per dag or
> globally
>>> * Per DAG multi processing: More robust and faster DAG processing. A
>>> faulty DAG should not take down the scheduler any more
>>> * Google Cloud Operators: Feature complete I have heard.
>>> * Time units now dynamic UI
>>> * Better SMTP handling and attachment support
>>> * Operational metrics for the scheduler
>>> * MSSQL Improvements
>>> * Experimental Rest API with Kerberos support
>>> * Auto alignment of start_date to interval

Re: Good jinja templating in airflow examples

2017-01-18 Thread Boris Tyukin 
Hi Guilherme,

I guess it depends what exactly you want to do as not everything works with
jinja.

>From documentation:

https://pythonhosted.org/airflow/concepts.html#jinja-templating

You can use Jinja templating with every parameter that is marked as
“templated” in the documentation.

You can open source code for operators and see what parameters are actually
templated.

For example, if you open source code for BashOperator
https://pythonhosted.org/airflow/_modules/bash_operator.html#BashOperator
you will see this line of code:
template_fields = ('bash_command', 'env')

It means only bash_command and env will work with jinja templates.

On Wed, Jan 18, 2017 at 12:45 PM, Guilherme Marthe <
guilherme.mar...@enjoei.com.br> wrote:

> Hey folks!
>
> I am trying to write a dag that works well with jinja templating, since
> through my study of the documentation, is the best way to ensure
> compatibility with the back-fill function.
>
> Are you guys aware of any examples online with this functionality working?
> The, documentation is still a bit dry on insightful examples, and I am a
> noobie developer, so I am trying to make the best use of the tool in the
> "correct way".
>
> Thank you in advance for any resources you guys can share!
>
> Gui
>
> Ps: I ve been told I can email you guys with questions like these :) Hope
> this is not a hassle ^_^
>