Re: AppVeyor for Windows compatibility testing

Hi, On Jenkins, we can target Windows executor (already available). Probably the easiest way. Regards JB On 10/19/2016 11:54 PM, Lukasz Cwik wrote: I noticed that the Maven exec plugin was using AppVeyor to get testing to occur on windows. Since this is currently a gap in our coverage today,

Re: Release Guide

Hi, well done. As already discussed, it looks good to me ;) Regards JB On 10/20/2016 01:24 AM, Davor Bonaci wrote: Hi everybody, As a project, I think we should have a Release Guide to document the process, have consistent releases, on-board additional release managers, and generally share

Placement of temporary files by FileBasedSink

Hello, This is a continuation of the discussion on PR https://github.com/apache/incubator-beam/pull/1050 which turned out more complex than expected. Short summary: Currently FileBasedSink, when writing to /path/to/foo (in practice, /path/to/foo-x-of-y where y is the total number of

Re: AppVeyor for Windows compatibility testing

I think we should use Apache Jenkins to get this coverage. It supports both cross-platform and cross-JDK coverage. It should be relatively straightforward to get this enabled. On Wed, Oct 19, 2016 at 2:54 PM, Lukasz Cwik wrote: > I noticed that the Maven exec plugin

Release Guide

Hi everybody, As a project, I think we should have a Release Guide to document the process, have consistent releases, on-board additional release managers, and generally share knowledge. It is also one of the project graduation guidelines. Dan and I wrote a draft version, documenting the process

AppVeyor for Windows compatibility testing

I noticed that the Maven exec plugin was using AppVeyor to get testing to occur on windows. Since this is currently a gap in our coverage today, is this something we can enable much like our Travis CI for the Apache Beam project?

Jenkins build is back to normal : beam_Release_NightlySnapshot #204

See

Re: [DISCUSS] Sources and Runners

On Wed, Oct 19, 2016 at 11:00 AM, Kenneth Knowles wrote: > I wanted to attempt to explicitly answer Raghu's question by saying that I > think Dan's starting points imply that the recommended behavior for start() > and advance() is to be "non-blocking" in the sense that

Re: [DISCUSS] Sources and Runners

Hi FYI when working on IO I already setup a docker image that I'm using for integration test. The IO unit tests embed and bootstrap the IO resources when possible. For instance JmsIO unit tests start a embedded ActiveMQ broker. However I also have a ActiveMQ docker image that I use for

Re: [DISCUSS] Sources and Runners

Hadoop FS has the local file system implementation that can be used for testing ("file" URL, no service needed). Thanks On Wed, Oct 19, 2016 at 10:43 AM, Amit Sela wrote: > Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka > native is best. > I'm

Re: [DISCUSS] Sources and Runners

It will be very useful for existing KafkaIOTest as well. MockConsumer we use is too primitive. ~ 50% of KafkaIOTest deals with MockConsumer. On Wed, Oct 19, 2016 at 10:43 AM, Amit Sela wrote: > Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka > native

Re: Simplifying User-Defined Metrics in Beam

On Thu, Oct 13, 2016 at 2:27 AM Aljoscha Krettek wrote: I finally found the time to have a look. :-) The API looks very good! (It's very similar to an API we recently added to Flink, which is inspired by the same Codahale/Dropwizard metrics). About the semantics, the

Re: [DISCUSS] Sources and Runners

Oh cool, that didn't exist in 0.8 I think, but anything that is Kafka native is best. I'm pretty sure there's an embedded HDFS for testing as well. While embedded Kafka/HDFS won't reflect "real-life" distributed environment, it could be a good place to start and provide some basic functional

Re: [jira] [Commented] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

At the point in time this was created, there were `NeedsRunner` tests. On Wed, Oct 19, 2016 at 9:34 AM, Kenneth Knowles (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/BEAM-755?page=com. > atlassian.jira.plugin.system.issuetabpanels:comment- >

Re: [DISCUSS] Sources and Runners

The SparkRunner actually has an embedded Kafka for its unit tests. On Wed, Oct 19, 2016, 20:16 Thomas Weise wrote: > Kafka can be embedded for the integration testing, which should > significantly simplify the setup. > > Here is an example I found: > >

Re: [DISCUSS] Sources and Runners

My thoughts: * It's worth reading the Beam testing document that Jason Kuster wrote! * Beam already has support for "End-to-end" integration tests, of examples (e.g., WordCountIT