Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

On Sat, Oct 20, 2018 at 12:24 AM Kenneth Knowles wrote: > Pinging this. I think Beam should have a live LTS branch. > > I want to suggest a different approach: choose something already released > to be LTS. This way it has had some usage and we have some confidence there > are no critical

Re: a new contributor

Hi Heejong, Thanks for introducing yourself! Welcome :) -Max On 19.10.18 21:45, Ankur Goenka wrote: Welcome Heejong! On Fri, Oct 19, 2018 at 12:27 PM Rui Wang > wrote: Welcome! -Rui On Fri, Oct 19, 2018 at 11:55 AM Robin Qiu

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

Hi Rui, The seed job being broken is sort of a big deal because it prevents updates to our Jenkins jobs. However, it doesn't stop the existing test configurations from running. I haven't found the mails annoying but I'm ok with moving them to the builds@ list. -Max On 22.10.18 11:05, Colm

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

We had a discussion recently about splitting the JIRA notifications to a new list "issues@b.a.o", and also splitting the Jenkins mails to (potentially) "builds@b.a.o". So I guess the issue you raised can be done once the latter mailing list is set up and active. Colm. On Fri, Oct 19, 2018 at

Re: [ANNOUNCE] New committers, October 2018

Congrats Ankur and Xinyu! On 19.10.18 21:27, Rui Wang wrote: Congrats and thanks for your contributions! -Rui On Fri, Oct 19, 2018 at 11:55 AM Ahmet Altay > wrote: Congratulations to both of you! :) On Fri, Oct 19, 2018 at 11:52 AM, Robin Qiu

Re: [PROPOSAL] Move sorting to sdks-java-core

What should be the next step? I guess we all agree that hadoop dependency should be splitted out. Then we're left off with the SortValues transform + in memory implementation. I'm ok with keeping this as a separate module, as this would discourage users to use sorting in their business logic.

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

Ah good point. That's why we were frequently seeing these mails. Would be nice to be able to test Jenkins DSL changes without generating an email to a mailing list. On 22.10.18 17:25, Kenneth Knowles wrote: Another thing about the seed job: for jobs like 'Java Postcommit" there is a separate

Re: Docker missing on Beam15

I've seen the docker issue pop-up on website pre-commits as well: https://issues.apache.org/jira/browse/BEAM-5783. There were also on beam15. When I searched around the internet I found lots of instances of the same error; it seems to be some unreliability in the guts of Docker [1]. Perhaps

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

I took a peek and the seed job seemed a bit different than the test jobs that use PostCommitJobBuilder to construct their normal & _PR variants. I didn't see how to appropriately plug in the helper code after 5 minutes of reading. But I am probably just a bit unfamiliar with the refactoring and

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

Yes, the SeedJob is not ideal because it's the only way to test Jenkins job changes, but there's no easy way to sandbox your testing-- running the seed job regenerates all live jobs from your PR. Splitting the SeedJob like we have for others based on trigger (cron vs phrase) would be an easy

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

OK, bui...@beam.apache.org now exists. I've opened https://github.com/apache/beam/pull/6775 to migrate but first make the discussion more specific. There are three types of notifications that are treated differently: - postcommits, etc, which were going to commits@ - nightly release failures

Re: Python docs build error

Great, thanks! Out of curiosity, did the jenkins job for the initial PR not detect the build failure? Colm. On Mon, Oct 22, 2018 at 2:29 PM Maximilian Michels wrote: > Correction for the footnote: > > [1] https://github.com/apache/beam/pull/6637 > > On 22.10.18 15:24, Maximilian Michels wrote:

Re: [DISCUSS] Move beam_SeedJob notifications to another email address

Another thing about the seed job: for jobs like 'Java Postcommit" there is a separate job for whether it is run as an actual postcommit or whether it is run against a PR. The seed job has not been split this way. So when someone is testing the seed job on a PR failures look the same as if it is

Build failed in Jenkins: beam_SeedJob #2842

See -- GitHub pull request #6789 of commit 3ee472615c01896e463da9277f3a798e647ac62f, no merge conflicts. Setting status of 3ee472615c01896e463da9277f3a798e647ac62f to PENDING with url

Build failed in Jenkins: beam_SeedJob #2843

See -- GitHub pull request #6789 of commit c68080f6dae08b617803c434eadff4073ac28555, no merge conflicts. Setting status of c68080f6dae08b617803c434eadff4073ac28555 to PENDING with url

Jenkins build is back to normal : beam_SeedJob #2844

See

Re: [VOTE] Release 2.8.0, release candidate #1

You two did so much verification I had a hard time finding something where my help was meaningful! :-) I did run the Nexmark suite on the DirectRunner against 2.7.0 and 2.8.0 following https://beam.apache.org/documentation/sdks/java/nexmark/#running-smoke-suite-on-the-directrunner-local . It is

Re: Python docs build error

Correction for the footnote: [1] https://github.com/apache/beam/pull/6637 On 22.10.18 15:24, Maximilian Michels wrote: Hi Colm, This [1] got merged recently and broke the "docs" target which apparently is not part of our Python PreCommit tests. See the following PR for a fix:

Re: 2 tier input

Chaim - If the full list of IDs is able to fit comfortably in memory and the Mongo collection is small enough that you can read the whole collection, you may want to fetch the IDs into a Java collection using the BigQuery API directly, then turn them into a Beam PCollection using

Beam Dependency Check Report (2018-10-22)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue google-cloud-bigquery 0.25.0 1.6.0

Re: [DISCUSS] Publish vendored dependencies independently

+1 for publishing vendored Jars independently. It will improve build time and ease IntelliJ integration. Flink also publishes shaded dependencies separately: - https://github.com/apache/flink-shaded - https://issues.apache.org/jira/browse/FLINK-6529 AFAIK their main motivation was to get rid

Python docs build error

Hi all, The following command: ./gradlew :beam-sdks-python:docs gives me the following error: /home/coheig/src/apache/beam/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py:docstring of

Re: Python docs build error

Hi Colm, This [1] got merged recently and broke the "docs" target which apparently is not part of our Python PreCommit tests. See the following PR for a fix: https://github.com/apache/beam/pull/6774 Best, Max [1] https://github.com/apache/beam/pull/6737 On 22.10.18 12:55, Colm O

Re: [PROPOSAL] Move sorting to sdks-java-core

Lukasz, you are right. I didn't think about structured coders. Thanks On Mon, Oct 22, 2018 at 7:40 PM Lukasz Cwik wrote: > I don't believe an interface will work because KvCoder/ListCoder/... would > only be order preserving if their components coders were order preserving. > > On Mon, Oct 22,

Re: [PROPOSAL] Move sorting to sdks-java-core

A related approach to Robert's that does not involve new types is to alter coder inference from the current: 1. Ask for a coder for type T 2. Check that the coder is (order preserving / deterministic) To: 1. Ask for an order preserving coder for T / ask for a deterministic coder for T This

Re: Java PreCommit and PostCommit cannot be triggered from PR

It starts to recover. Tests have been triggered for most of pending PRs. -Rui On Mon, Oct 22, 2018 at 11:27 AM Rui Wang wrote: > I am facing the same issue. > > It might be related to instability of Github: > https://status.github.com/messages. > > -Rui > > On Mon, Oct 22, 2018 at 11:22 AM

Re: Java PreCommit and PostCommit cannot be triggered from PR

Thanks for the update! On Mon, Oct 22, 2018 at 12:56 PM Rui Wang wrote: > It starts to recover. Tests have been triggered for most of pending PRs. > > -Rui > > On Mon, Oct 22, 2018 at 11:27 AM Rui Wang wrote: > >> I am facing the same issue. >> >> It might be related to instability of Github:

Re: Python docs build error

To Colm's question. We observed this issue as well and had discussions in a separate thread , with Scott and Micah. This issue was only reproduced on certain Linux environment. MacOS does not have this error. We also specifically ran the test on

Re: Docker missing on Beam15

Thanks Yifan! On Mon, Oct 22, 2018 at 2:53 PM Yifan Zou wrote: > So, looks like none of us have the permissions. I filed INFRA-17167 > to the Infra team to > restart the docker on the beam15. > > Thanks. > Yifan > > On Mon, Oct 22, 2018 at

Re: What is required for LTS releases? (was: [PROPOSAL] Prepare Beam 2.8.0 release)

On Mon, Oct 22, 2018 at 2:11 AM, Robert Bradshaw wrote: > On Sat, Oct 20, 2018 at 12:24 AM Kenneth Knowles wrote: > >> Pinging this. I think Beam should have a live LTS branch. >> >> I want to suggest a different approach: choose something already released >> to be LTS. This way it has had some

Re: Docker missing on Beam15

So, looks like none of us have the permissions. I filed INFRA-17167 to the Infra team to restart the docker on the beam15. Thanks. Yifan On Mon, Oct 22, 2018 at 9:20 AM Scott Wegner wrote: > I've seen the docker issue pop-up on website

Java PreCommit and PostCommit cannot be triggered from PR

Hey, It seems like the Java PreCommit and Java PostCommit cannot be triggered from PR from this morning. Does anyone have the same issue or have any idea what happened? Boyuan Zhang

Re: Java PreCommit and PostCommit cannot be triggered from PR

I am facing the same issue. It might be related to instability of Github: https://status.github.com/messages. -Rui On Mon, Oct 22, 2018 at 11:22 AM Boyuan Zhang wrote: > Hey, > > It seems like the Java PreCommit and Java PostCommit cannot be triggered > from PR from this morning. Does anyone

Re: [PROPOSAL] Move sorting to sdks-java-core

I don't believe an interface will work because KvCoder/ListCoder/... would only be order preserving if their components coders were order preserving. On Mon, Oct 22, 2018 at 8:52 AM David Morávek wrote: > What should be the next step? I guess we all agree that hadoop dependency > should be

Re: [VOTE] Release 2.8.0, release candidate #1

Hi all, Did you have a chance to review this RC? Between me and Robert we ran a significant chunk of the validations. Let me know if you have any questions. Ahmet On Thu, Oct 18, 2018 at 5:26 PM, Ahmet Altay wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the

Re: [DISCUSS] Publish vendored dependencies independently

OK, I've filed https://issues.apache.org/jira/browse/BEAM-5819 to collect sub-tasks. This has enough upsides throughout lots of areas of the project that even though it is not glamorous it seems pretty valuable to start on immediately. And I want to find out if there's a pitfall lurking. Max:

Re: Docker missing on Beam15

Looks like we have more container build related errors. This is from beam6 - https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/44/ Reading package lists... [91mW: The repository 'http://deb.debian.org/debian stretch Release' does not have a Release file. W: The repository