Re: [VOTE] First stable release: release candidate #4

+1 (binding) Tested on beam-samples, especially focus on HDFS support, etc. Thanks ! Regards JB On 05/13/2017 06:47 AM, Davor Bonaci wrote: Hi everyone -- After going through several release candidates, setting and validating acceptance criteria, running a hackathon, and polishing the release,

Re: [VOTE] First stable release: release candidate #4

+1 -- validated several common cases on spark yarn and dataflow runner. Exciting milestone for the project! On May 12, 2017 21:48, "Davor Bonaci" wrote: Hi everyone -- After going through several release candidates, setting and validating acceptance criteria, running a hackathon, and polishing

[PROPOSAL] design of DSL SQL interface

Hi all, As you may know, we're working on BeamSQL to execute SQL queries as a Beam pipeline. This is a valuable feature, not only shipped as a packaged CLI, but also as part of the SDK to assemble a pipeline. I prepare a document[1] to list the high level APIs, to show how SQL queries can be adde

Re: First stable release: release candidate #3

This RC is now obsolete -- see RC #4 information with a formal vote in a separate thread [1]. Davor [1] https://lists.apache.org/thread.html/8cf5d7583111f7d67bd21918eb35a29823572ad9ebc1d6df9656bc79@%3Cdev.beam.apache.org%3E On Fri, May 12, 2017 at 12:57 AM, Davor Bonaci wrote: > The release ca

[VOTE] First stable release: release candidate #4

Hi everyone -- After going through several release candidates, setting and validating acceptance criteria, running a hackathon, and polishing the release, now is the time to vote! Please review and vote on the release candidate #4 for the version 2.0.0, as follows: [ ] +1, Approve the release [ ]

Re: Towards a spec for robust streaming SQL, Part 1

Tyler, Yes, dynamic table changes over time. You can find more details about dynamic table from this Flink blog ( https://flink.apache.org/news/2017/04/04/dynamic-tables.html). Fabian, me and Xiaowei posted it a week before the flink-forward@SF. "A dynamic table is a table that is continuously upda

Re: First stable release: Acceptance criteria

Validated same examples Vikas mentioned for Windows. Updated the doc on acceptance criteria. Thanks, Cham On Fri, May 12, 2017 at 4:43 PM Vikas RK wrote: > Just validated wordcount and mobile gaming examples for Python SDK on > Direct and Dataflow runner. Mostly looks good to me, with minor cha

Re: First stable release: Acceptance criteria

Just validated wordcount and mobile gaming examples for Python SDK on Direct and Dataflow runner. Mostly looks good to me, with minor changes that could be done to improve user experience, but not a blocker for FSR. (filed BEAM-2286 ) On 12 May 2017

Re: [PROPOSAL] Apache Hive connector

Hi! Why do you need at all to override methods like computeSplitsIfNecessary - is HCatalogIO substantially different from other HadoopInputFormat's that it can not be handled by the generic code of HadoopInputFormatIO? I looked at the implementation in your commit and it seems identical, except fo

Re: [PROPOSAL] Apache Hive connector

Hi Eugene, In order to reuse HadoopInputFormatIO, this is what I am thinking - 1. Extend HadoopInputFormatBoundedSource to create - HCatalogBoundedSource 2. Override necessary methods in HCatalogBoundedSource to perform HCatalog-specific steps. ( overriding computeSplitsIfNecessary() method shoul

Re: Towards a spec for robust streaming SQL, Part 1

Being able to support an EMIT config independent of the query itself sounds great for compatible use cases (which should be many :-). Shaoxuan, can you please refresh my memory what a dynamic table means in Flink? It's basically just a state table, right? The "dynamic" part of the name is to simpl

Re: [DISCUSSION] using NexMark for Beam

I think these are valuable enough that we should get them into apache/master On Fri, May 12, 2017 at 4:34 AM, Jean-Baptiste Onofré wrote: > Hi, > > PR or even a feature branch could work. Up to you. > > Regards > JB > > > On 05/12/2017 10:55 AM, Etienne Chauchot wrote: > >> Hi guys, >> >> I want

Re: TextIO and .withWindowedWrites() - filenamepolicy

Can we simply fail WWW if windowed writes is not set? Or at least warn? On May 12, 2017 6:14 PM, "Dan Halperin" wrote: DefaultFilenamePolicy as currently written only accepts a single shard name template. if that template is windowed, it won't work for unwindowed writes (it will have -WWW-PPP or

Re: TextIO and .withWindowedWrites() - filenamepolicy

yep... I figured that myself.. let me work with JB and others to see if we can come up with satisfactory default implementation On Fri, 12 May 2017 at 16:14 Dan Halperin wrote: > DefaultFilenamePolicy as currently written only accepts a single shard name > template. if that template is windowed

Re: TextIO and .withWindowedWrites() - filenamepolicy

DefaultFilenamePolicy as currently written only accepts a single shard name template. if that template is windowed, it won't work for unwindowed writes (it will have -WWW-PPP or .WWW.PPP or something.WWW.something.?). And vice versa on unwindowed templated for windowed writes. There is likely

Re: Pull request - power function

Thanks @Tarush, will also take a look. On Fri, May 12, 2017 at 7:19 AM, Jean-Baptiste Onofré wrote: > Thanks, > > we gonna take a look. > > Regards > JB > > > On 05/12/2017 04:12 PM, tarush grover wrote: > >> Hi Team, >> >> I have opened a pull request Beam-2171 power function #3092. >> >> Kindl

Re: Pull request - power function

Thanks, we gonna take a look. Regards JB On 05/12/2017 04:12 PM, tarush grover wrote: Hi Team, I have opened a pull request Beam-2171 power function #3092. Kindly review and verify. Regards, Tarush -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.t

Pull request - power function

Hi Team, I have opened a pull request Beam-2171 power function #3092. Kindly review and verify. Regards, Tarush

Re: First stable release: Acceptance criteria

Enjoy Kenn ! Regards JB On 05/12/2017 03:38 PM, Kenneth Knowles wrote: I wanted to let this thread & list know that I'll be offline camping starting ~now through the weekend. So if you see any open thread with me on it, please do take it over. Things are looking pretty good, so I might not see

Re: First stable release: Acceptance criteria

I wanted to let this thread & list know that I'll be offline camping starting ~now through the weekend. So if you see any open thread with me on it, please do take it over. Things are looking pretty good, so I might not see any more action. I have a lot of faith in this community to validate the re

Re: [DISCUSSION] using NexMark for Beam

Hi, PR or even a feature branch could work. Up to you. Regards JB On 05/12/2017 10:55 AM, Etienne Chauchot wrote: Hi guys, I wanted to let you know that I have just submitted a PR around NexMark. This is a port of the NexMark queries to Beam, to be used as integration tests. This can also be

Re: TextIO and .withWindowedWrites() - filenamepolicy

Hi Reuven, yes, it's what I have in mind: just provide the default logic in the existing method. Regards JB On 05/12/2017 12:46 PM, Reuven Lax wrote: DefaultFilenamePolicy already contains a windowedFilename override (today it throws an exception), so I don't think there's any need for a new

Re: TextIO and .withWindowedWrites() - filenamepolicy

DefaultFilenamePolicy already contains a windowedFilename override (today it throws an exception), so I don't think there's any need for a new class. We can simply fill out the existing method. On May 12, 2017 11:34 AM, "Borisa Zivkovic" wrote: +1 for DefaultFilenamePolicy being able to understa

Re: Towards a spec for robust streaming SQL, Part 1

Thanks to Tyler and Fabian for sharing your thoughts. Regarding to the early/late update control of FLINK. IMO, each dynamic table can have an EMIT config. For FLINK table-API, this can be easily implemented in different manners, case by case. For instance, in window aggregate, we could define "wh

Re: [DISCUSSION] using NexMark for Beam

Hi guys, I wanted to let you know that I have just submitted a PR around NexMark. This is a port of the NexMark queries to Beam, to be used as integration tests. This can also be used as A-B testing (no-regression or performance comparison between 2 versions of the same engine or of the same r

Re: TextIO and .withWindowedWrites() - filenamepolicy

Great... created this https://issues.apache.org/jira/browse/BEAM-2276 On Fri, 12 May 2017 at 09:38 Jean-Baptiste Onofré wrote: > +1 > > Borisa, if you want, we can work together on this. > > Thanks ! > Regards > JB > > On 05/12/2017 10:33 AM, Borisa Zivkovic wrote: > > +1 for DefaultFilenameP

Re: TextIO and .withWindowedWrites() - filenamepolicy

+1 Borisa, if you want, we can work together on this. Thanks ! Regards JB On 05/12/2017 10:33 AM, Borisa Zivkovic wrote: +1 for DefaultFilenamePolicy being able to understand basic windowing... probably the most user-friendly way that would cover most of needs... in case of special needs user

Re: TextIO and .withWindowedWrites() - filenamepolicy

+1 for DefaultFilenamePolicy being able to understand basic windowing... probably the most user-friendly way that would cover most of needs... in case of special needs users can provide their own policy.. another alternative would be to have new class called DefaultWindowedFilenamePolicy in packa

Re: Towards a spec for robust streaming SQL, Part 1

2017-05-11 7:14 GMT+02:00 Tyler Akidau : > On Tue, May 9, 2017 at 3:06 PM Fabian Hueske wrote: > > > Hi Tyler, > > > > thank you very much for this excellent write-up and the super nice > > visualizations! > > You are discussing a lot of the things that we have been thinking about > as > > well f

Re: First stable release: release candidate #2

This RC is now obsolete -- see RC #3 information in a separate thread [1]. Davor [1] https://lists.apache.org/thread.html/25493fded77367299ad185c308d658eeb10e4f03be725154b0e5b08d@%3Cdev.beam.apache.org%3E On Wed, May 10, 2017 at 10:28 PM, Davor Bonaci wrote: > The release candidate #2 for the

First stable release: release candidate #3

The release candidate #3 for the version 2.0.0 has been built. I think this candidate could be it -- we have cleared the entire list of blocking changes [8]! I'd like to ask everyone to give this candidate a try -- this is the last opportunity we have to fix any critical issues if the final relea

Re: TextIO and .withWindowedWrites() - filenamepolicy

I believe that for most windows there is a standard stringification. However I think we could allow the user to inject a window formatter for cases where there is no good default (e.g. where the window is a complicated user-defined type, and toString() isn't good enough. Alternatively, if we don't