Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #3

See Changes: [rezarokni] initial commit for BEAM-2871. WIP Todo: Add in the unit tests to work [relax] Update python version. [relax] Update container version. [kirpichov] Cherrypick of #3995

Re: Gradle Status [April 11]

Found another blocker in current artifacts creations: there is not pom.xml and pom.properties in META-INF. This is used by tools + libraries + integrations so it is quite important to not break it. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-12 0:59 GMT+

Re: Gradle Status [April 6]

My test was also on version 2017, 2017.2.7 to be precise Etienne Le mercredi 11 avril 2018 à 17:01 +, Daniel Oliveira a écrit : > Hi everyone, I was the one who initially wrote the PR with Idea instructions. > I was using 2017.3 as well while writing > it so all the instructions were tested on

Re: [PROPOSAL] Preparing 2.5.0 release next week

+1 to what Ahmet said,  I also think that it is important to take our time given that we're in the gradle migration process. +1 to what JB said also: try gradle and fall back to maven in case of troubles. Etienne Le mercredi 11 avril 2018 à 13:35 -0700, Ahmet Altay a écrit : > +1 to delaying 2 wee

Re: Gradle Status [April 11]

Nice ! thanks Kenn Le mercredi 11 avril 2018 à 18:21 +, Kenneth Knowles a écrit : > Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner + Netty > has been introduced since yesterday. > Etienne mentioned he has worked toward setting up periodic runs on all > runners, so this

Re: Gradle Status [April 6]

Daniel, actually I did run it with default IDEA JUnit test runner. Then, in “Settings > Build, Execution, Deployment > Build Tools > Gradle > Runner" I selected “Gradle Test Runner” in “Run tests using” selectbox and it works ok when I run my tests with IDEA shortcuts. So, probably, we should ad

grade test setup different than maven one?

Hi guys, did the gradle track changed the way test execution was done? This PR https://github.com/apache/beam/pull/4965 works very well with maven and sometimes doesn't pass with gradle. Think we should keep the previous setup which was globally reliable (I'm not speaking of tests which are not b

Re: Running Nexmark with PubSub issue

Raghu, Kenneth, Yes, creating a separate class instead of inner one helped to overcome this issue with serialisation. Seems like this a bug in NexmarkLauncher, so I’ll create a jira for this. Thank you for help with this. Btw, Raghu, are you going to submit a PR from your branch? I think this i

Re: [PROPOSAL] Preparing 2.5.0 release next week

Romain, could you please open a JIRA describing the requirements for the generated pom's? The gradle-generated pom's don't match the maven version byte-for-byte, but I don't think that's a requirement. I and others are still hacking on the Gradle build, so it's possible we could get the pom's read

Re: Gradle Status [April 11]

> Found another blocker in current artifacts creations: there is not pom.xml and pom.properties in META-INF. This is used by tools + libraries + integrations so it is quite important to not break it Romain, is there a JIRA for this issues? If not could you create one please? On Thu, Apr 12, 2018

Re: [PROPOSAL] Preparing 2.5.0 release next week

created https://issues.apache.org/jira/browse/BEAM-4057 Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-12 18:11 GMT+02:00 Scott Wegner : > Romain, could you please open a JIRA describing the requirements for the > generated pom's? The gradle-generated pom's

Re: grade test setup different than maven one?

It looks like the precommit failure [1] is for a new test that was added. Have you debugged the test to ensure it's not flaky? [1] https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4059/testReport/junit/org.apache.beam.runners.direct/ExecutorServiceParallelExecutorTest/ensureMetricsThr

Re: Gradle Status [April 11]

just created one https://issues.apache.org/jira/browse/BEAM-4057 Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-12 18:15 GMT+02:00 Ahmet Altay : >> Found another blocker in current artifacts creations: there is not > pom.xml and pom.properties in META-INF.

Re: [PROPOSAL] Preparing 2.5.0 release next week

Just because I have probably missed it in the flood of work/emails (yay!) have we been dry-running the user experience of a Maven-built project depending on Beam? If declaring a dependency on Beam doesn't work in some basic way, it seems like we would want to know that right away. Our release verif

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Sounds perfect. Just wanted to make sure that "custom metrics of supported type" didn't include new ways of aggregating ints. As long as that means we have a fixed set of aggregations (that align with what what users want and metrics back end support) it seems like we are doing user metrics right.

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Agree with all of this. It echoes a thread on the doc that I was going to bring here. Let's keep it simple and use concrete use cases to drive additional abstraction if/when it becomes compelling. Kenn On Thu, Apr 12, 2018 at 9:21 AM Ben Chambers wrote: > Sounds perfect. Just wanted to make sur

Re: grade test setup different than maven one?

on maven it is quite reliable, ran it > 10 times without any failure. I suspect (but didnt check by lack of time) gradle parallelism is different somehow and can lead to some flackyness here. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book 2018-04-12 18:20 GMT+02:0

Re: grade test setup different than maven one?

It seems that the test probably depends on some details of Maven or our Maven configuration. If so, that's a problem with the test. It should be able to succeed in any build system, or as a standalone JUnit main built from the suite, etc. Kenn On Thu, Apr 12, 2018 at 9:27 AM Romain Manni-Bucau w

Beam7 Outage

Hi all, The Jenkins Beam7 executor gave up the ghost some time in the last couple of days. I've been on the line with Infra yesterday and today getting it fixed, and it looks like it should be back up in a few hours. I'll ping this thread again when I have confirmation. Thanks for your patience.

Re: grade test setup different than maven one?

Seems so but is not. Most of beam tests assume they are alone in their vm when executing for a bunch of reason, if not the case a lot of side effects can happen (backend state, local cache drop, , uncontrolled resources and failure due to the GBKTest which creates 100M keys etc...) so you have

Re: Gradle Status [April 6]

I think it depends on what exactly switching to "Gradle Test Runner" from "Platform Test Runner" does. I tried it out on my machine and they seem to act identically to each other. The IntelliJ documentation says it determines what API to use to run the tests

Re: Gradle Status [April 6]

@Daniel: did you restart in between? Otherwise it does nothing. One launches JunitCoreRunner from idea and the other a gradle command. Le 12 avr. 2018 19:24, "Daniel Oliveira" a écrit : > I think it depends on what exactly switching to "Gradle Test Runner" from > "Platform Test Runner" does. I t

Re: Gradle Status [April 6]

Ah, I did not. Thanks Romain. I tried it again, restarting in between, and still had no differences. Since it seems like there's no reason not to use "Gradle Test Runner", I'll mention it in the contributor's guide. On Thu, Apr 12, 2018 at 10:31 AM Romain Manni-Bucau wrote: > @Daniel: did you r

Re: Beam7 Outage

beam5 has been failing in the last week. Almost all builds there break. On Thu, Apr 12, 2018, 6:41 PM Jason Kuster wrote: > Hi all, > > The Jenkins Beam7 executor gave up the ghost some time in the last couple > of days. I've been on the line with Infra yesterday and today getting it > fixed, an

Re: Beam7 Outage

I can ask Infra if there's an issue with that machine as well, but if it's still accessible at all it's probably not the same issue; jenkins couldn't get to beam7 at all. On Thu, Apr 12, 2018 at 12:11 PM Ismaël Mejía wrote: > beam5 has been failing in the last week. Almost all builds there break

Re: Gradle Status [April 6]

Well you are the only one to not have the drawbacks to use it so maybe dont do it? I know Luke is in holidays but anyone else with the knowledge of why we nees that noise compared to idea native tooling/flow? Le 12 avr. 2018 20:16, "Daniel Oliveira" a écrit : > Ah, I did not. Thanks Romain. > >

Re: Beam7 Outage

They all seem flaky over the past few days. I just hit one on beam1: java.io.IOException: Backing channel 'beam1' is disconnected. https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4068/console Could there be some load issue from the Gradle changes? Andrew On Thu, Apr 12, 2018 at 1

Re: Splittable DoFN in Spark discussion

(resurrecting thread as I'm back from leave) I looked at this mode, and indeed as Reuven points out it seems that it affects execution details, but doesn't offer any new APIs. Holden - your suggestions of piggybacking an unbounded-per-element SDF on top of an infinite stream would work if 1) there

Re: Gradle Status [April 6]

Sorry Romain, I'm not quite sure what you're asking. Can you clarify? On Thu, Apr 12, 2018 at 12:22 PM Romain Manni-Bucau wrote: > Well you are the only one to not have the drawbacks to use it so maybe > dont do it? I know Luke is in holidays but anyone else with the knowledge > of why we nees t

Re: Gradle Status [April 6]

I also don't quite understand what your question is, and it appears like Dan spent considerable time trying to reproduce your issue. For the record, I have had no issues running tests via Gradle in IntelliJ for the past few weeks. Reuven On Thu, Apr 12, 2018 at 9:47 PM Daniel Oliveira wrote: >

Add a (temporary) Portable Flink branch to the ASF repo?

Hey all, I've been working on a proof-of-concept portable Flink runner with some other Beam contributors. We would like to have a point of reference for the rest of the Beam community as we integrate this work into master. It currently lives under https://github.com/bsidhom/beam/tree/hacking-job-

Re: Add a (temporary) Portable Flink branch to the ASF repo?

How much of this is not suitable to merging into master directly (not as is, but as separate PRs)? On Thu, Apr 12, 2018 at 3:10 PM Ben Sidhom wrote: > Hey all, > I've been working on a proof-of-concept portable Flink runner with some other Beam contributors. We would like to have a point of refe

Re: Add a (temporary) Portable Flink branch to the ASF repo?

I would say that *most* of it is not suitable for direct merging. There are several reasons for this: - Most changes are built on upstream PRs that are either not submitted or have been rebased before submission. - There are some very hacky changes in the Python and Java SDKs to get po

Re: Add a (temporary) Portable Flink branch to the ASF repo?

I would also be in favour of adding a branch to our main repo. A random branch on some personal GitHub account can seem a bit sketchy and adding a branch to our repo could make it more visible for people that are interested. > On 12. Apr 2018, at 15:29, Ben Sidhom wrote: > > I would say that

Re: Add a (temporary) Portable Flink branch to the ASF repo?

I suppose with the hackathon and flink forward behind us, I'm thinking we should start shifting gears more getting what we have into master in production state and less on continuing working on a hacking branch. If we think it'll fairly quick there's no big need to create an official branch, and if

Re: Add a (temporary) Portable Flink branch to the ASF repo?

So I would be strongly in favour of adding it as a branch on the Apache repo. This way other folks are more likely to be able to help with the splitting up and merging process and also while Flink forward is behind us getting in the practice of doing feature branches on the ASF repo for collaborati

Re: Add a (temporary) Portable Flink branch to the ASF repo?

Strong +1 on transitioning all development to the ASF repo. I think a straight move of the hacking branch may still be problematic though, because it sets the path to continue hacking vs. working towards a viable milestone that other contributors can base their work off. I would prefer a state tha

Re: Add a (temporary) Portable Flink branch to the ASF repo?

+ 1 to capture in JIRA what needs to be done. The simplest path forward might be to reimplement/cherrypick'n'modify the changes onto master directly. We would then effectively just abandon the hacking branch and treat code there as a prototype. Although we would add components without end2end veri

Re: Add a (temporary) Portable Flink branch to the ASF repo?

Thomas captures exactly my concerns. I think we should look at getting everything we can into master (at least filing JIRAs, the work may take longer) and move what development we can there. What remains would be a collection of hacks (mostly in the SDKs, but it's not a feature branch 'cause we'd

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

I agree that there is some confusion about concepts. Here are several concepts which have come up in discussions, as I see them (not official names). *Metric* - For the purposes of my document, I have been referring to a Metric as any sort of information the SDK can send to the Runner

Re: Add a (temporary) Portable Flink branch to the ASF repo?

Maybe add a toggle in flinkoptions to activate it until it is tested and you are happy with it? Kind of --flinkExperimental=sdf,log,... (random values). This allows to hit master and continue to work on it. Le 13 avr. 2018 03:07, "Robert Bradshaw" a écrit : Thomas captures exactly my concerns.

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

On Thu, Apr 12, 2018 at 8:17 PM Alex Amato wrote: > I agree that there is some confusion about concepts. Here are several > concepts which have come up in discussions, as I see them (not official > names). > > *Metric* > >- For the purposes of my document, I have been referring to a Metric >

Re: Gradle Status [April 6]

When you launch a test with gradle runner it launches gradle which makes loose 3s on a very fast computer and more on a slower (6 on my personal one which is already fast but not as much as my work one). We are 5 to see that regression at least. So there is a reason to not use the gradle runner if