Re: Beam SQL Pipeline Options

2018-06-13 Thread Andrew Pilloud
I've turned this into a PR, more discussion going on over there: https://github.com/apache/beam/pull/5592 Andrew On Wed, Jun 6, 2018 at 9:46 PM Kenneth Knowles wrote: > This is a nice short design discussion doc, and perhaps a cooler piece of > news hidden in the paragraph :-) > > Kenn > > On

Re: Beam SQL Pipeline Options

2018-06-07 Thread Kenneth Knowles
Hi Arun, If you are looking to connect to a SQL database from Beam Java code, then I you might want JdbcIO [1] with a PostgreSQL JDBC driver [2]. This is not currently wired up to Beam SQL specially; you have to connect in Java code. It would be a nice contribution. Kenn [1]

Re: Beam SQL Pipeline Options

2018-06-06 Thread arun kumar
Hi Thanks for the update. Can you please share me if you have any documentation for connecting to postgres using beam code. Thanks Arun On Wed, Jun 6, 2018, 9:54 PM Andrew Pilloud wrote: > We are just about to the point of having a working pure SQL workflow for > Beam! One of the last things

Re: Beam SQL Pipeline Options

2018-06-06 Thread Kenneth Knowles
This is a nice short design discussion doc, and perhaps a cooler piece of news hidden in the paragraph :-) Kenn On Wed, Jun 6, 2018 at 9:24 AM Andrew Pilloud wrote: > We are just about to the point of having a working pure SQL workflow for > Beam! One of the last things that remains is how to

Beam SQL Pipeline Options

2018-06-06 Thread Andrew Pilloud
We are just about to the point of having a working pure SQL workflow for Beam! One of the last things that remains is how to configure Pipeline Options via a SQL shell. I have written up a proposal to use the set statement, for example "SET runner=DataflowRunner". I'm looking for feedback,