Re: compileJava broken on master see: BEAM-6495

After your last message, Alex, I saw the issue on a branch with minimal unrelated changes on top of b4b5495307 (January 28). I ran `./gradlew clean` for the first time in a while at that time, and worked happily for

Re: [PROPOSAL] Prepare Beam 2.11.0 release

+1 let's keep things going. Thanks for volunteering! Andrew On Thu, Feb 7, 2019, 4:49 PM Kenneth Knowles wrote: > I think this is a good idea. Even though 2.10.0 is still making it's way > out, there's still 6 weeks of changes between the cut dates, lots of value. > It is good to keep the

Re: [PROPOSAL] Prepare Beam 2.11.0 release

I think this is a good idea. Even though 2.10.0 is still making it's way out, there's still 6 weeks of changes between the cut dates, lots of value. It is good to keep the rhythm and follow the calendar. Kenn On Thu, Feb 7, 2019, 15:52 Ahmet Altay Hi all, > > Beam 2.11 release branch cut date

Re: Another another new contributor! :)

Welcome to the community Kyle! On Thu, Feb 7, 2019 at 1:22 AM Alexey Romanenko wrote: > Welcome, Kyle! > Great to have more people on SparkRunner side! > > Btw, feel free to join “beam-spark” channel on Slack. > > Alexey > > On 7 Feb 2019, at 08:44, Reza Ardeshir Rokni wrote: > > Welcome! > >

JIRA priorities explaination

Hello Beam community, I was thinking about this and found some information to share/discuss. Would it be possible to confirm my thinking on this: - There are 5 priorities in the JIRA system today (tooltip link

[PROPOSAL] Prepare Beam 2.11.0 release

Hi all, Beam 2.11 release branch cut date is 2/13 according to the release calendar [1]. I would like to volunteer myself to do this release. I intend to cut the branch on the planned 2/13 date. If you have releasing blocking issues for 2.11 please mark their "Fix Version" as 2.11.0. (I also

Re: Jenkins slowness

Migration to gradle 5 should not be an issue. Unless something unforeseen happens as we did not run all possible task yet. Should only be a merge of an pending + one additional PR upgrading the versions. > On 7. Feb 2019, at 21:23, Udi Meiri wrote: > > I suggest disabling Jacoco and

Re: Jenkins slowness

I saw many non-beam jobs are running on our nodes. It probably be one of the reasons which caused long waiting time. e.g. https://builds.apache.org/computer/beam13/builds On Thu, Feb 7, 2019 at 12:24 PM Udi Meiri wrote: > There is also excessive python test logging tracked here: >

Re: Jenkins slowness

I suggest disabling Jacoco and re-enabling the build cache until we can migrate to Gradle 5. I imagine the migration to v5 is not a simple change. Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run postcommit). On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath Seems like there

Re: Jenkins slowness

There is also excessive python test logging tracked here: https://issues.apache.org/jira/browse/BEAM-6603 On Thu, Feb 7, 2019, 12:23 Udi Meiri I suggest disabling Jacoco and re-enabling the build cache until we can > migrate to Gradle 5. I imagine the migration to v5 is not a simple change. >

Re: Jenkins slowness

Seems like there was a spike for all build times yesterday probably added up to give slow Jenkins scheduling times for triggers. Also, seems like we had three spikes that are about a week apart recently. On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey wrote: > What might have some influence is

Re: [BEAM-6594] Flakey GrpcDataServiceTest.testMessageReceivedBySingleClientWhenThereAreMultipleClients - failing in precommit

This was already reported in BEAM-6512 [1], which Scott gave me as a starter bug. I haven't been able to reproduce locally, so I'm trying to see if I can get it to fail on Jenkins again with some additional logging [2]. Definitely interested in other's thoughts on this, I only vaguely understand

Re: Jenkins slowness

What might have some influence is the implicit disabling of the build cache by activating Jacoco report. There seems to be a increase of beam_PreCommit_Java_Cron with https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and looking into cacheable task there seems to be lots of work done now

Jenkins slowness

If anyone has done any investigation/is working on this please share. I'm investigating Jenkins slowness. I've noticed it happening since yesterday: precommits taking 3 hours to start, phrase commands similarly taking as much time to register. My current theory is that we have a job that's are

Re: Jenkins slowness

Precommits times for Python and Java have been slowly climbing: http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1=1546974237153=1549566237153=light On Thu, Feb 7, 2019 at 10:54 AM Udi Meiri wrote: > If anyone has done any investigation/is working on this please share. > > I'm

Re: [VOTE] Release 2.10.0, release candidate #3

+1 I validated running: * Java Quickstart (Direct) * Java Quickstart (Apex local) * Java Quickstart (Flink local) * Java Quickstart (Spark local) * Java Quickstart (Dataflow) * Java Mobile Game (Dataflow) On Wed, Feb 6, 2019 at 2:28 PM Kenneth Knowles wrote: > Hi everyone, > > Please review

Re: [VOTE] Release 2.10.0, release candidate #3

+1 I tested briefly new HadoopFormatIO (read/write), deprecated HadoopInputFormatIO (read) and new KafkaIO feature (write into multiple topics) > On 7 Feb 2019, at 12:01, Maximilian Michels wrote: > > +1 (binding) > > On 06.02.19 23:47, Reuven Lax wrote: >> +1 (binding) >> On Wed, Feb 6,

Re: pipeline steps

Shouldn't this apply to more generic scenario for any BeamIO? For example, I am using KafkaIO and wanted to get the topic and partition from which the message was received. Some IOContext associated with each data unit from BeamIO may be useful here? -Yi On Thu, Feb 7, 2019 at 6:29 AM Kenneth

Re: pipeline steps

This comes up a lot, wanting file names alongside the data that came from the file. It is a historical quirk that none of our connectors used to have the file names. What is the change needed for FileIO + parse Avro to be really easy to use? Kenn On Thu, Feb 7, 2019 at 6:18 AM Jeff Klukas

Re: Installing Apache Beam locally

It is a bit of a funky invocation, and our config makes it a little stranger yet. I think you want ./gradlew -Ppublishing -PnoSigning publishMavenJavaPublicationToMavenLocal -Ppublishing is in our config to enable/disable the publishing plugin -PnoSigning turns off signing which is on by

Re: pipeline steps

I haven't needed to do this with Beam before, but I've definitely had similar needs in the past. Spark, for example, provides an input_file_name function that can be applied to a dataframe to add the input file as an additional column. It's not clear to me how that's implemented, though. Perhaps

pipeline steps

Hi, I am working on a pipeline that listens to a topic on pubsub to get files that have changes in the storage. Then i read avro files, and would like to write them to bigquery based on the file name (to different tables). My problem is that the transformer that reads the avro does not give me

Re: [VOTE] Release 2.10.0, release candidate #3

+1 (binding) On 06.02.19 23:47, Reuven Lax wrote: +1 (binding) On Wed, Feb 6, 2019 at 2:28 PM Kenneth Knowles > wrote: Hi everyone, Please review and vote on the release candidate #3 for the version 2.10.0, as follows: [ ] +1, Approve the release

Re: Another another new contributor! :)

Welcome, Kyle! Great to have more people on SparkRunner side! Btw, feel free to join “beam-spark” channel on Slack. Alexey > On 7 Feb 2019, at 08:44, Reza Ardeshir Rokni wrote: > > Welcome! > > On Tue, 5 Feb 2019 at 23:34, Kenneth Knowles > wrote: > Welcome Kyle! >

Re: Another another new contributor! :)

Hi, Help much appreciated !And welcome ! Etienne Le jeudi 07 février 2019 à 15:44 +0800, Reza Ardeshir Rokni a écrit : > Welcome! > On Tue, 5 Feb 2019 at 23:34, Kenneth Knowles wrote: > > Welcome Kyle! > > On Tue, Feb 5, 2019 at 4:34 AM Maximilian Michels wrote: > > > Welcome Kyle! Excited to