Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

Looks like these are getting sent to dev. Is there a per-project config we need to tweak or a ticket we need to file in order to get these sent to commits@ instead? On Mon, Aug 13, 2018 at 6:17 PM GitBox wrote: > aaltay commented on issue #1: Add .gitmodules file > URL:

[GitHub] boyuanzz opened a new pull request #2: Add travis build config files and upload artifacts into dist.apache.org

boyuanzz opened a new pull request #2: Add travis build config files and upload artifacts into dist.apache.org URL: https://github.com/apache/beam-wheels/pull/2 Add there config files to enable python wheels build with travis. How to trigger travis build from this kind of

[GitHub] aaltay commented on issue #2: Add travis build config files and upload artifacts into dist.apache.org

aaltay commented on issue #2: Add travis build config files and upload artifacts into dist.apache.org URL: https://github.com/apache/beam-wheels/pull/2#issuecomment-412946929 @boyuanzz Please add the instruction to a README type of a file. It will be easier for people to find it.

Re: Implementing @OnWindowExpiration in StatefulParDo [BEAM-1589]

I can review it. On Tue, Aug 14, 2018 at 10:26 AM Huygaa Batsaikhan wrote: > Finally, I have a PR for the > annotation itself. Anyone up for reviewing it? Ken has been helping me, but > he is going to be OOO for a while. > > On Tue, Mar 20, 2018 at

External resources in Beam pipelines

Hi folks - Many of you don't know me, as I don't contribute directly to Beam. But I do a lot of work around the periphery, in particular considering how to manage and monitor Beam pipelines. I think there's room in Beam to greatly improve both the management and monitoring story, especially

[GitHub] boyuanzz opened a new pull request #3: Add multibuild submodule

boyuanzz opened a new pull request #3: Add multibuild submodule URL: https://github.com/apache/beam-wheels/pull/3 Add multibuild submodule into root project. Following up with PR https://github.com/apache/beam-wheels/pull/1 r: @aaltay

Re: External resources in Beam pipelines

Here's the raw doc content, for your convenience.Beam I/O representation In the current Beam programming model, sources and sinks are virtually indistinguishable from other transforms. From a composition point of view, this is great. But from an integration point of view, special handling of

Re: Test failures list

Hi everyone, We have increased amount of test jobs failures recently. In terms of numbers (based on my memory and http://35.226.225.164/): Java precommits went down from ~55% to ~30% of succeeded jobs. Java postcommits went down from ~60 to ~40 of succeeded jobs. I'm currently triaging

Re: Implementing @OnWindowExpiration in StatefulParDo [BEAM-1589]

Finally, I have a PR for the annotation itself. Anyone up for reviewing it? Ken has been helping me, but he is going to be OOO for a while. On Tue, Mar 20, 2018 at 4:23 PM Huygaa Batsaikhan wrote: > As echauchot@ mentioned, it will make it easier and

[GitHub] boyuanzz commented on issue #2: Add travis build config files and upload artifacts into dist.apache.org

boyuanzz commented on issue #2: Add travis build config files and upload artifacts into dist.apache.org URL: https://github.com/apache/beam-wheels/pull/2#issuecomment-412951558 Re: @aaltay There are some bug needed fixes. After everything is ok, I'll add README and ping you to review.

[GitHub] aaltay closed pull request #3: Add multibuild submodule

aaltay closed pull request #3: Add multibuild submodule URL: https://github.com/apache/beam-wheels/pull/3 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a

Re: [Discuss] Add EXTERNAL keyword to CREATE TABLE statement

Thanks Mikhail! "Import" is an alternative option. It might be better. "create external" is being widely used by different systems with similar meaning so "create" usually is ok to external data sources. -Rui On Tue, Aug 14, 2018 at 9:38 AM Mikhail Gryzykhin wrote: > The idea of clarification

Re: [Discuss] Add EXTERNAL keyword to CREATE TABLE statement

The idea of clarification sounds good to me. I'd appreciate that present, when I was triaging post-commit tests. Do we have any terms that specify connection to external table? "CREATE" word triggers this reaction in my brain that there will be a new table created. Adding "EXTERNAL" would already

Re: [HELP] Blog post for 2.6.0 Release

Great! Thank you for fixing this. > On 13 Aug 2018, at 21:20, Rafael Fernandez wrote: > > Howdy! https://github.com/apache/beam-site/pull/536 > should take care of this. > > Cheers, > r > > > On Mon, Aug 13, 2018 at 9:14 AM Austin Bennett

Re: Policy for Python ValidatesRunner vs IT tests?

The ValidatesContainer test builds the python portable worker container using docker, pushes it, then runs a dataflow pipeline using the container with the flag --worker_harness_container_image. There is only one test that does this, and it is also an 'IT' test. On Tue, Aug 14, 2018 at 3:27 PM

Re: Policy for Python ValidatesRunner vs IT tests?

Hi Pablo, Beam Java has unit tests, ValidatesRunner tests, and Integration (E2E) test. If I understand correctly, unit tests test the functionality of a component and usually don't require a runner (is this true?); ValidatesRunner tests (marked with @ValidatesRunner annotation) require runners;

Re: Policy for Python ValidatesRunner vs IT tests?

Ah this is very helpful. I think this is missing in the testing guide. I'll make a short PR to mention it. Best -P. On Tue, Aug 14, 2018 at 3:37 PM Anton Kedin wrote: > IT tests exist in java, similar to unit tests and not marked in a special > way, except they're called *IT.java instead of

Re: Rename Nexmark jobs to Perf instead of PostCommit

@Reuven Lax Thank you for clarifying. It makes sense to keep them as post-commits then. @Ahmet Altay I don't see them being flaky atm. So we are good here. I would prefer to separate perf tests mostly because of requirements, such as runtime, environment, trigger restrictions. Right now we

Re: Policy for Python ValidatesRunner vs IT tests?

IT tests exist in java, similar to unit tests and not marked in a special way, except they're called *IT.java instead of *Test.java. They're run from corresponding tasks: -

Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

It's probably an INFRA ticket to change the mailing list preferences. On Tue, Aug 14, 2018 at 8:47 AM Boyuan Zhang wrote: > Hey Jason, > > I'm going to figure out this problem today. Do you have idea what should > we do or where should we start? > > Boyuan > > On Tue, Aug 14, 2018 at 1:15 AM

Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

Hey all, beam-wheels has sent messages to commits@. Thanks you dealing with the INFRA ticket, Ahmet! Boyuan Zhang On Tue, Aug 14, 2018 at 3:55 PM Jason Kuster wrote: > It's probably an INFRA ticket to change the mailing list preferences. > > On Tue, Aug 14, 2018 at 8:47 AM Boyuan Zhang

[Build Python Wheels] New Repository for Building Python Wheels

Hey all, Now we have beam-wheels as a wrapper repository to build and stage python wheels for each release. In order to stage all artifacts successfully, it requires apache credentials. Please take a look at README

Policy for Python ValidatesRunner vs IT tests?

Hello, In Python, we tag some test methods with @attr('ValidatesRunner') and @attr('IT'), which marks them to be run as pipeline tests. If I understand correctly: - ValidatesRunner tests are more like a component tests[1] as explained in Beam docs - IT tests are more like a E2E test[2] as

Re: [Discussion] Clarify the support story for released Beam versions

Still waiting for any additional user feedback to come. I added reviewers to the PR. Unless there is objections or additional feedback I would like to go ahead with this version as it is. Modifications after that would always be welcome. On Mon, Aug 13, 2018 at 2:06 PM, Rafael Fernandez wrote:

Re: [Build Python Wheels] New Repository for Building Python Wheels

Thanks Boyuan : D - this is very nice to have. On Tue, Aug 14, 2018 at 4:30 PM Boyuan Zhang wrote: > Hey all, > > Now we have beam-wheels as a > wrapper repository to build and stage python wheels for each release. In > order to stage all artifacts

Beam Dependency Check Report (2018-08-15)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue High Priority Dependency Updates Of Beam Java SDK:

Publishing release artifacts to custom artifactory

Hi, I wanted to see if anyone has used the Gradle build to publish the release/snapshot artifacts to custom repo? So far I found the following (attempting to publish a particular Maven artifact): ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing

Re: [GitHub] aaltay commented on issue #1: Add .gitmodules file

Hey Jason, I'm going to figure out this problem today. Do you have idea what should we do or where should we start? Boyuan On Tue, Aug 14, 2018 at 1:15 AM Jason Kuster wrote: > Looks like these are getting sent to dev. Is there a per-project config we > need to tweak or a ticket we need to