Re: Dynamic file-based sinks

2017-06-08 Thread Reuven Lax
After looking at everyone's comments, I think option 1 is the better approach - map destinations to a FilenamePolicy. It is a good parallel to what we do in BigQueryIO (the main difference is that we're mapping to a sharded filename, instead of a single destination like in BigQueryIO). The main li

Re: Windows OS Compatibility and Jenkins Postcommit

2017-06-08 Thread Flavio Fiszman
Thanks Luke! I'm working on fixing the Windows incompatibilities related to file schemes and globs in FileSystems. On Thu, Jun 8, 2017 at 3:52 PM, Lukasz Cwik wrote: > I added the Windows Jenkins Postcommit run to be able to allow for people > to address a class of issues when attempting to buil

Windows OS Compatibility and Jenkins Postcommit

2017-06-08 Thread Lukasz Cwik
I added the Windows Jenkins Postcommit run to be able to allow for people to address a class of issues when attempting to build, test, run the SDK with the Windows OS. It is not expected that the Jenkins Postcommit passes or will pass until the issues below are addressed. I was hoping that contribu

Re: [jira] [Created] (BEAM-2426) Remove imports from runner/init

2017-06-08 Thread Robert Bradshaw
Yeah, wish we had gotten to this pre-alpha. On Thu, Jun 8, 2017 at 3:10 PM, Ahmet Altay (JIRA) wrote: > Ahmet Altay created BEAM-2426: > - > > Summary: Remove imports from runner/init > Key: BEAM-2426 > URL: https://is

Re: Apache Beam SQL: DDL

2017-06-08 Thread Tyler Akidau
Thanks James, added some initial comments. -Tyler On Thu, Jun 8, 2017 at 3:11 AM James wrote: > Hi team, I am working on BeamSQL now, and would like to add DDL for > BeamSQL, so prepared a design doc[1], would like to hear your opinions :) > > [1] > > https://docs.google.com/document/d/162_cuYl

Apache Beam SQL: DDL

2017-06-08 Thread James
Hi team, I am working on BeamSQL now, and would like to add DDL for BeamSQL, so prepared a design doc[1], would like to hear your opinions :) [1] https://docs.google.com/document/d/162_cuYlZ5pC_8PzGWX844tlLOsSvQmoSGjJAgol4ipE/edit?usp=sharing

Re: [DISCUSS] Source Watermark Metrics

2017-06-08 Thread JingsongLee
Hi @Ben Chambers @Aljoscha Krettek @Aviem Zur and other all,  I've written this up as a proposal found here:  https://docs.google.com/document/d/1ykjjG97DjVQP73jGbotGRbtK38hGvFbokNEOuNO4DAo/edit?usp=sharing Feel free to comment/edit it. Best, JingsongLee --