Re: [PROPOSAL] Prepare Beam 2.6.0 release

2018-07-30 Thread Pablo Estrada
Hello everyone, we will explore a workaround for the issues with the Dataflow worker. Because of this, the 2.6.0 release is no longer blocked by the Dataflow workers at the moment. I hope to produce a release candidate by end-of-day tomorrow. Please reach out on this thread, or to me directly if

Beam Dependency Check Report (2018-07-31)

2018-07-30 Thread Apache Jenkins Server
High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release google-cloud-bigquery 0.25.0 1.4.0 2017-06-26

Beam Dependency Check Report (2018-07-31)

2018-07-30 Thread Apache Jenkins Server
High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release google-cloud-bigquery 0.25.0 1.4.0 2017-06-26

Re: SQS source

2018-07-30 Thread John Rudolf Lewis
I created a pr for my SqsIO contribution. I look forward to your comments. https://github.com/apache/beam/pull/6101 Any chance this could be a part of the 2.6.0 release? On Thu, Jul 19, 2018 at 7:39 AM, John Rudolf Lewis wrote: > Thank you. > > I've created a jira ticket to add SQS and have

Re: delayed emit (timer) in py-beam?

2018-07-30 Thread Austin Bennett
Fantastic; thanks, Charles! On Mon, Jul 30, 2018 at 3:49 PM, Charles Chen wrote: > Hey Austin, > > This API is not yet implemented in the Python SDK. I am working on this > feature: the next step from my end is to finish a reference implementation > in the local DirectRunner. As you note,

Re: delayed emit (timer) in py-beam?

2018-07-30 Thread Charles Chen
Hey Austin, This API is not yet implemented in the Python SDK. I am working on this feature: the next step from my end is to finish a reference implementation in the local DirectRunner. As you note, the doc at https://s.apache.org/beam-python-user-state-and-timers describes the design. You

delayed emit (timer) in py-beam?

2018-07-30 Thread Austin Bennett
What's going on with timers and python? Am looking at building a pipeline (assuming another group in my company will grant access to the Kafka topic): Kafka -> beam -> have beam wait 24 hours -> do transform(s) and emit a record. If I read things correctly that's not currently possible in

Re: Live coding & reviewing adventures

2018-07-30 Thread Holden Karau
So small schedule changes. I’ll be doing some poking at the Go SDK at 2pm today - https://www.youtube.com/watch?v=9UAu1DOZJhM and the one with Gris setting up Beam on a new machine will be moved to Friday because her laptop got delayed - https://www.youtube.com/watch?v=x8Wg7qCDA5k On Tue, Jul 24,

Re: Beam Docs Contributor

2018-07-30 Thread Thomas Weise
Welcome Rose, and looking forward to the docs update! On Mon, Jul 30, 2018 at 9:15 AM Henning Rohde wrote: > Welcome Rose! Great to have you here. > > On Mon, Jul 30, 2018 at 2:23 AM Ismaël Mejía wrote: > >> Welcome ! >> Great to see someone new working in this important area for the project.

Re: Beam Docs Contributor

2018-07-30 Thread Henning Rohde
Welcome Rose! Great to have you here. On Mon, Jul 30, 2018 at 2:23 AM Ismaël Mejía wrote: > Welcome ! > Great to see someone new working in this important area for the project. > > > On Mon, Jul 30, 2018 at 5:57 AM Kai Jiang wrote: > >> Welcome Rose! >> ᐧ >> >> On Sun, Jul 29, 2018 at 8:53 PM

Re: Issues with Beam SQL on Spark

2018-07-30 Thread Andrew Pilloud
That sounds great. I'm subscribed to that list as well, so I'll keep an eye out for your email. Andrew On Sun, Jul 29, 2018 at 9:07 PM Kai Jiang wrote: > Hi Andrew, > > I tried on replacing "jdbc:calcite" to "jdbc:beam" in calcite and > re-shadow. After that, Beam Sql can run on Spark now. >

Beam Dependency Check Report (2018-07-30)

2018-07-30 Thread Apache Jenkins Server
High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release google-cloud-bigquery 0.25.0 1.4.0 2017-06-26

Re: ElasticsearchIO bulk delete

2018-07-30 Thread Tim Robertson
> we decided to postpone the feature That makes sense. I believe the ES6 branch is in-part working (I've looked at the code but not used it) which you can see here [1] and the jira to watch or contribute is [2]. It would be a useful addition to test independently and report any observations or

Re: Beam Docs Contributor

2018-07-30 Thread Ismaël Mejía
Welcome ! Great to see someone new working in this important area for the project. On Mon, Jul 30, 2018 at 5:57 AM Kai Jiang wrote: > Welcome Rose! > ᐧ > > On Sun, Jul 29, 2018 at 8:53 PM Rui Wang wrote: > >> Welcome! >> >> -Rui >> >> On Sun, Jul 29, 2018 at 7:07 PM Griselda Cuevas wrote: >>