Re: Broken seed job

[Łukasz Gajowy]

I didn't edit the "Standalone Seed job", only the "SeedJob". Now every time
someone tries to run the seed job ("Run seed job") it results in an error
even despite prior running the standalone job from master branch the way
you described.



śr., 11 lip 2018 o 01:25 Lukasz Cwik  napisał(a):

> job_seed_standalone should only be edited when we know that the regular
> seed job is in a healthy state.
>
> Note, that you can always recover back to what is checked in master by:
> 1) Creating an empty PR
> 2) Using the standalone seed job trigger phrase: "Run Standalone Seed Job"
>
> I kicked one off right now:
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob_Standalone/1289/
>
> On Tue, Jul 10, 2018 at 4:17 PM Łukasz Gajowy 
> wrote:
>
>> Hi,
>>
>> while working on Jenkins' seed job, some changes I introduced didn't get
>> reverted even after running the seed job again from the master branch. This
>> is why seed job is now failing. More details here [1] and here [2].
>>
>> Since I can operate only with jobs phrase-triggered from GitHub's PR, I
>> think there's nothing more I can do than I already tried. In my
>> opinion, fixing this issue requires an aid of a person with some greater
>> access to Jenkins. Can someone help with that?
>>
>> Sorry for the inconvenience - I didn't expect that such situation can
>> occur. "job_seed_standalone" works fine and it can be used instead
>> (until the issue is fixed).
>>
>> [1] https://github.com/apache/beam/pull/5915
>> [2]
>> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob/2190/console
>>
>>
>> Best regards,
>> Łukasz
>>
>

Build failed in Jenkins: beam_SeedJob #2193

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit d647a14cb3aa72e3b21e5a94f1430b00245a84b2, 
no merge conflicts.
Setting status of d647a14cb3aa72e3b21e5a94f1430b00245a84b2 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2193/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision f65ecb8b7fd195d02352862ab993565b35986fbb 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f65ecb8b7fd195d02352862ab993565b35986fbb
Commit message: "Merge d647a14cb3aa72e3b21e5a94f1430b00245a84b2 into 
109f03c45af5fa287d938d696c95ef50dad2ab91"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy
Not sending mail to unregistered user ccla...@bluewin.ch

Re: Broken seed job

[Lukasz Cwik]

job_seed_standalone should only be edited when we know that the regular
seed job is in a healthy state.

Note, that you can always recover back to what is checked in master by:
1) Creating an empty PR
2) Using the standalone seed job trigger phrase: "Run Standalone Seed Job"

I kicked one off right now:
https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob_Standalone/1289/

On Tue, Jul 10, 2018 at 4:17 PM Łukasz Gajowy 
wrote:

> Hi,
>
> while working on Jenkins' seed job, some changes I introduced didn't get
> reverted even after running the seed job again from the master branch. This
> is why seed job is now failing. More details here [1] and here [2].
>
> Since I can operate only with jobs phrase-triggered from GitHub's PR, I
> think there's nothing more I can do than I already tried. In my
> opinion, fixing this issue requires an aid of a person with some greater
> access to Jenkins. Can someone help with that?
>
> Sorry for the inconvenience - I didn't expect that such situation can
> occur. "job_seed_standalone" works fine and it can be used instead (until
> the issue is fixed).
>
> [1] https://github.com/apache/beam/pull/5915
> [2]
> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob/2190/console
>
>
> Best regards,
> Łukasz
>

Broken seed job

[Łukasz Gajowy]

Hi,

while working on Jenkins' seed job, some changes I introduced didn't get
reverted even after running the seed job again from the master branch. This
is why seed job is now failing. More details here [1] and here [2].

Since I can operate only with jobs phrase-triggered from GitHub's PR, I
think there's nothing more I can do than I already tried. In my
opinion, fixing this issue requires an aid of a person with some greater
access to Jenkins. Can someone help with that?

Sorry for the inconvenience - I didn't expect that such situation can
occur. "job_seed_standalone" works fine and it can be used instead (until
the issue is fixed).

[1] https://github.com/apache/beam/pull/5915
[2]
https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob/2190/console

Best regards,
Łukasz

Build failed in Jenkins: beam_SeedJob #2192

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2192/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision a1b830fc5a67572aaa50a011d7621429805635d2 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a1b830fc5a67572aaa50a011d7621429805635d2
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
109f03c45af5fa287d938d696c95ef50dad2ab91"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy
Not sending mail to unregistered user ccla...@bluewin.ch

[PROPOSAL] JIRA Automation for Beam Dependency Upgrades

[Yifan Zou]

Hi all,

I have a short proposal on automatically file JIRA tickets that tracks the
Beam dependency update process. Any comments are welcome.

https://docs.google.com/document/d/1XXTMnofizSQZSorZy4NFlxKfx5f17PGRmws-DngoTms/edit#

Regards.
Yifan

Build failed in Jenkins: beam_SeedJob #2191

[Apache Jenkins Server]

See 

--
GitHub pull request #5920 of commit 4715449665ae235f96c7a67ce5f80b774d5474e5, 
no merge conflicts.
Setting status of 4715449665ae235f96c7a67ce5f80b774d5474e5 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2191/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5920/*:refs/remotes/origin/pr/5920/*
 > git rev-parse refs/remotes/origin/pr/5920/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5920/merge^{commit} # timeout=10
Checking out Revision f7ef62db30f8422dad1e614380a0c21ff268dc5c 
(refs/remotes/origin/pr/5920/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f7ef62db30f8422dad1e614380a0c21ff268dc5c
Commit message: "Merge 4715449665ae235f96c7a67ce5f80b774d5474e5 into 
109f03c45af5fa287d938d696c95ef50dad2ab91"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy
Not sending mail to unregistered user ccla...@bluewin.ch

Vendoring / Shading Protobuf and gRPC

[Lukasz Cwik]

With the merge of PR #5594[1], we started shading all gRPC / Protobuf
dependencies within all the modules that depended on the model/*
dependencies by vendoring them. The vendored versions are built and
packaged into the model jars (they should be separated out once I figure
out how to generate proto code using a shaded import path). Note that this
cleaned up several issues where we were incorrectly built shaded jars
without repackaging in some locations or the shading process was corrupting
the contents of some of the jars.

Note that the majority of the code base (especially related to portability)
should be using imports under the org.apache.beam.vendor.protobuf.v3 or
org.apache.beam.vendor.grpc.v1 paths. I have yet to figure out a clean way
to get Intellij to recognize these vendored paths. My only solution so far
has been to manually add one of the built model jars to the compile
classpath of the module being worked on in Intellij as described here[2]. I
would greatly appreciate some ideas on how to improve this integration
because from a few attempts configuring the intellij gradle pluglin scope
sections didn't produce the result that I was expecting.

I also added a simple test task
*validateShadedJarDoesntLeakNonOrgApacheBeamClasses* that validates the
shaded jar doesn't contain classes without repackaging which we should
apply to any module that performs shading to ensure that classes are
relocated and we don't accidentally expose stuff. I filed BEAM-4753[3] to
this end.

Note, these are the following modules that still depend on protobuf that
are shaded away and could move to use a vendored variant of protobuf:
* sdks/java/core
* sdks/java/extensions/sql

These modules expose protobuf because it is part of the API surface:
* sdks/java/extensions/protobuf
* sdks/java/io/google-cloud-platform (I believe that gRPC could be shaded
here but preferrably the IO module would do it so we wouldn't have this
maintenance burden.)

1: https://github.com/apache/beam/pull/5594
2:
https://stackoverflow.com/questions/1051640/correct-way-to-add-external-jars-lib-jar-to-an-intellij-idea-project
3: https://issues.apache.org/jira/browse/BEAM-4753

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

Robert, the docs  say:
"The people you choose as code owners must have write permissions for the
repository,"
but I want to ignore that for now and see what happens. :)

Also, I'm not sure how the auto-assignment works. Does it round-robin? Will
it assign more than one reviewer?
We'll see how it goes.

Feel free to add components as you see fit. Most of what I've done is
guesswork.

On Tue, Jul 10, 2018 at 11:13 AM Ahmet Altay  wrote:

> +1
>
> I added my name and a few others names that frequently do reviews in some
> areas.
>
> On Tue, Jul 10, 2018 at 10:23 AM, Jean-Baptiste Onofré 
> wrote:
>
>> +1
>>
>> I added my name on some components ;)
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 10/07/2018 02:06, Udi Meiri wrote:
>> > Hi everyone,
>> >
>> > I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
>> > mechanism.
>> > Initial version is here: _
>> https://github.com/apache/beam/pull/5909/files_
>> >
>> > I need help from the community in determining owners for each component.
>> > Feel free to directly edit the PR (if you have permission) or add a
>> comment.
>> >
>> >
>> > Background
>> > The idea is to:
>> > 1. Document good review candidates for each component.
>> > 2. Help choose reviewers using the auto-assignment mechanism. The
>> > suggestion is in no way binding.
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Build failed in Jenkins: beam_SeedJob #2190

[Apache Jenkins Server]

See 


Changes:

[Pablo] Improving signing of all published artifacts and adding test publication

[Pablo] Removing extra publication

[cclauss] [BEAM-3761] Define cmp() in Python 3

[cclauss] [BEAM-3959] Add Python 3 undefined names to flake8

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4c81d4e7bc669680693300e9b94a6d93719232e8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4c81d4e7bc669680693300e9b94a6d93719232e8
Commit message: "Merge pull request #5843 from cclauss/Define-cmp-in-Python3"
 > git rev-list --no-walk 6f6401d7b30f9095ba9bd581301be0dbae3e24fe # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy
Not sending mail to unregistered user ccla...@bluewin.ch

Re: drop nexmark perfkit tables

[Etienne Chauchot]

Thanks  Andrew !Le mardi 10 juillet 2018 à 09:11 -0700, Andrew Pilloud a écrit :
> Done. The UI doesn't make it easy to drop the tables without dropping the 
> dataset. In case it needs to be done again,
> here is what I ran:
> for table in $(bq ls -a --dataset_id apache-beam-testing:nexmark | awk 'NR > 
> 2 {print $1}');
> do bq rm -f apache-beam-testing:nexmark.$table;
> done
> 
> Andrew
> On Tue, Jul 10, 2018 at 8:12 AM Etienne Chauchot  wrote:
> > Hi Guys,
> > 
> > Can anyone who has the rights (PMC members maybe) drop all the BigQuery 
> > tables from project `apache-beam-testing` in
> > the dataset `nexmark`, they contain incorrect data. 
> > 
> > Please do not drop the nexmark dataset, only the tables. Don't worry the 
> > tables will be re-created with next commit
> > on master.
> > 
> > Thanks
> > 
> > Etienne

Re: CODEOWNERS for apache/beam repo

[Ahmet Altay]

+1

I added my name and a few others names that frequently do reviews in some
areas.

On Tue, Jul 10, 2018 at 10:23 AM, Jean-Baptiste Onofré 
wrote:

> +1
>
> I added my name on some components ;)
>
> Thanks !
> Regards
> JB
>
> On 10/07/2018 02:06, Udi Meiri wrote:
> > Hi everyone,
> >
> > I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
> > mechanism.
> > Initial version is here: _https://github.com/
> apache/beam/pull/5909/files_
> >
> > I need help from the community in determining owners for each component.
> > Feel free to directly edit the PR (if you have permission) or add a
> comment.
> >
> >
> > Background
> > The idea is to:
> > 1. Document good review candidates for each component.
> > 2. Help choose reviewers using the auto-assignment mechanism. The
> > suggestion is in no way binding.
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: CODEOWNERS for apache/beam repo

[Rafael Fernandez]

+1!

On Tue, Jul 10, 2018 at 8:51 AM Robert Burke  wrote:

> +1
> If non-committers are welcome in the file, I'm happy to assist Henning
> with Go SDK reviews.(@lostluck)
>
>
> On Tue, Jul 10, 2018, 8:47 AM Alexey Romanenko 
> wrote:
>
>> +1,
>> Udi, thank you for taking care of this!
>> I added myself as a reviewer of some IO components.
>>
>>
>> On 10 Jul 2018, at 17:00, Henning Rohde  wrote:
>>
>> +1. Sounds like a useful improvement.
>>
>> Udi -- do the reviewers in this file need to be committers for the PR
>> auto-assignment to work?
>>
>> On Tue, Jul 10, 2018 at 1:59 AM Łukasz Gajowy 
>> wrote:
>>
>>> +1. It will certainly be useful. I added myself (and a fellow
>>> contributor) to some components (IO testing related mostly).
>>>
>>> Thanks,
>>> Łukasz
>>>
>>> wt., 10 lip 2018 o 02:06 Udi Meiri  napisał(a):
>>>
 Hi everyone,

 I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
 mechanism.
 Initial version is here: *https://github.com/apache/beam/pull/5909/files
 *

 I need help from the community in determining owners for each component.
 Feel free to directly edit the PR (if you have permission) or add a
 comment.


 Background
 The idea is to:
 1. Document good review candidates for each component.
 2. Help choose reviewers using the auto-assignment mechanism. The
 suggestion is in no way binding.



>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: drop nexmark perfkit tables

[Andrew Pilloud]

Done. The UI doesn't make it easy to drop the tables without dropping the
dataset. In case it needs to be done again, here is what I ran:

for table in $(bq ls -a --dataset_id apache-beam-testing:nexmark | awk 'NR
> 2 {print $1}');
do bq rm -f apache-beam-testing:nexmark.$table;
done

Andrew

On Tue, Jul 10, 2018 at 8:12 AM Etienne Chauchot 
wrote:

> Hi Guys,
>
> Can anyone who has the rights (PMC members maybe) drop all the BigQuery
> tables from project `apache-beam-testing` in the dataset `nexmark`, they
> contain incorrect data.
>
> Please do not drop the nexmark dataset, only the tables. Don't worry the
> tables will be re-created with next commit on master.
>
> Thanks
>
> Etienne
>

Build failed in Jenkins: beam_SeedJob #2189

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2189/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision 9048f7d095baac968a6c09d2fbbe852d507ecf90 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9048f7d095baac968a6c09d2fbbe852d507ecf90
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Re: CODEOWNERS for apache/beam repo

[Robert Burke]

+1
If non-committers are welcome in the file, I'm happy to assist Henning with
Go SDK reviews.(@lostluck)


On Tue, Jul 10, 2018, 8:47 AM Alexey Romanenko 
wrote:

> +1,
> Udi, thank you for taking care of this!
> I added myself as a reviewer of some IO components.
>
>
> On 10 Jul 2018, at 17:00, Henning Rohde  wrote:
>
> +1. Sounds like a useful improvement.
>
> Udi -- do the reviewers in this file need to be committers for the PR
> auto-assignment to work?
>
> On Tue, Jul 10, 2018 at 1:59 AM Łukasz Gajowy 
> wrote:
>
>> +1. It will certainly be useful. I added myself (and a fellow
>> contributor) to some components (IO testing related mostly).
>>
>> Thanks,
>> Łukasz
>>
>> wt., 10 lip 2018 o 02:06 Udi Meiri  napisał(a):
>>
>>> Hi everyone,
>>>
>>> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
>>> mechanism.
>>> Initial version is here: *https://github.com/apache/beam/pull/5909/files
>>> *
>>>
>>> I need help from the community in determining owners for each component.
>>> Feel free to directly edit the PR (if you have permission) or add a
>>> comment.
>>>
>>>
>>> Background
>>> The idea is to:
>>> 1. Document good review candidates for each component.
>>> 2. Help choose reviewers using the auto-assignment mechanism. The
>>> suggestion is in no way binding.
>>>
>>>
>>>
>

Re: CODEOWNERS for apache/beam repo

[Alexey Romanenko]

+1, 
Udi, thank you for taking care of this! 
I added myself as a reviewer of some IO components.

> On 10 Jul 2018, at 17:00, Henning Rohde  wrote:
> 
> +1. Sounds like a useful improvement.
> 
> Udi -- do the reviewers in this file need to be committers for the PR 
> auto-assignment to work?
> 
> On Tue, Jul 10, 2018 at 1:59 AM Łukasz Gajowy  > wrote:
> +1. It will certainly be useful. I added myself (and a fellow contributor) to 
> some components (IO testing related mostly). 
> 
> Thanks, 
> Łukasz
> 
> wt., 10 lip 2018 o 02:06 Udi Meiri  > napisał(a):
> Hi everyone,
> 
> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS 
> mechanism.
> Initial version is here: https://github.com/apache/beam/pull/5909/files 
> 
> 
> I need help from the community in determining owners for each component.
> Feel free to directly edit the PR (if you have permission) or add a comment.
> 
> 
> Background
> The idea is to:
> 1. Document good review candidates for each component.
> 2. Help choose reviewers using the auto-assignment mechanism. The suggestion 
> is in no way binding.
> 
> 

[metrics] inside the (not portable) runner

[Etienne Chauchot]

Guys,
 
I recently had the occasion to make a diagram to explain to a user the 
architecture of metrics inside the runner in the
not-portable world (for now).
I think I will use it for my upcoming talk at the ApacheCon. 
I wanted to share it with you as a piece of documentation and also for 
commenting/correcting.

https://docs.google.com/drawings/d/1BClBBHifaQhQwVnD5g8kHF0YT_eeAtoZaTmY3FFq8Co/edit?usp=sharing

Best
Etienne

Build failed in Jenkins: beam_SeedJob #2188

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2188/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision 9048f7d095baac968a6c09d2fbbe852d507ecf90 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9048f7d095baac968a6c09d2fbbe852d507ecf90
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Build failed in Jenkins: beam_SeedJob #2187

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2187/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision 9048f7d095baac968a6c09d2fbbe852d507ecf90 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9048f7d095baac968a6c09d2fbbe852d507ecf90
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Build failed in Jenkins: beam_SeedJob #2186

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2186/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision 9048f7d095baac968a6c09d2fbbe852d507ecf90 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9048f7d095baac968a6c09d2fbbe852d507ecf90
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Build failed in Jenkins: beam_SeedJob #2185

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2185/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision 9048f7d095baac968a6c09d2fbbe852d507ecf90 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9048f7d095baac968a6c09d2fbbe852d507ecf90
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Build failed in Jenkins: beam_SeedJob #2184

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2184/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision ccc430b1d8702346b52aea70ecd1933824281db9 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ccc430b1d8702346b52aea70ecd1933824281db9
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy

Build failed in Jenkins: beam_SeedJob #2183

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit 71ccdaa868e75409e9298dc8aef303022d7cd2c5, 
no merge conflicts.
Setting status of 71ccdaa868e75409e9298dc8aef303022d7cd2c5 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2183/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision 84e96206c80143bfc7a52b493db24ee9adf8b4ac 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 84e96206c80143bfc7a52b493db24ee9adf8b4ac
Commit message: "Merge 71ccdaa868e75409e9298dc8aef303022d7cd2c5 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
 > git rev-list --no-walk 84e96206c80143bfc7a52b493db24ee9adf8b4ac # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
Processing DSL script job_seed_standalone.groovy
Processing DSL script HadoopInputFormatJob.groovy
ERROR: startup failed:
HadoopInputFormatJob.groovy: 19: unable to resolve class 
common.CommonPropertiesFromDir
 @ line 19, column 1.
   import common.CommonPropertiesFromDir as commonProperties
   ^

1 error

drop nexmark perfkit tables

[Etienne Chauchot]

Hi Guys,

Can anyone who has the rights (PMC members maybe) drop all the BigQuery tables 
from project `apache-beam-testing` in the
dataset `nexmark`, they contain incorrect data. 

Please do not drop the nexmark dataset, only the tables. Don't worry the tables 
will be re-created with next commit on
master.

Thanks

Etienne

Re: CODEOWNERS for apache/beam repo

[Henning Rohde]

+1. Sounds like a useful improvement.

Udi -- do the reviewers in this file need to be committers for the PR
auto-assignment to work?

On Tue, Jul 10, 2018 at 1:59 AM Łukasz Gajowy 
wrote:

> +1. It will certainly be useful. I added myself (and a fellow contributor)
> to some components (IO testing related mostly).
>
> Thanks,
> Łukasz
>
> wt., 10 lip 2018 o 02:06 Udi Meiri  napisał(a):
>
>> Hi everyone,
>>
>> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
>> mechanism.
>> Initial version is here: *https://github.com/apache/beam/pull/5909/files
>> *
>>
>> I need help from the community in determining owners for each component.
>> Feel free to directly edit the PR (if you have permission) or add a
>> comment.
>>
>>
>> Background
>> The idea is to:
>> 1. Document good review candidates for each component.
>> 2. Help choose reviewers using the auto-assignment mechanism. The
>> suggestion is in no way binding.
>>
>>
>>

Build failed in Jenkins: beam_SeedJob #2181

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit 71ccdaa868e75409e9298dc8aef303022d7cd2c5, 
no merge conflicts.
Setting status of 71ccdaa868e75409e9298dc8aef303022d7cd2c5 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2181/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision 84e96206c80143bfc7a52b493db24ee9adf8b4ac 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 84e96206c80143bfc7a52b493db24ee9adf8b4ac
Commit message: "Merge 71ccdaa868e75409e9298dc8aef303022d7cd2c5 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
Processing DSL script job_seed_standalone.groovy
Processing DSL script HadoopInputFormatJob.groovy
ERROR: startup failed:
HadoopInputFormatJob.groovy: 19: unable to resolve class 
common.CommonPropertiesFromDir
 @ line 19, column 1.
   import common.CommonPropertiesFromDir as commonProperties
   ^

1 error

Build failed in Jenkins: beam_SeedJob #2180

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b, 
no merge conflicts.
Setting status of 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2180/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision 8800c4ae431a6379b0c9dbad04e31b49a325fe14 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8800c4ae431a6379b0c9dbad04e31b49a325fe14
Commit message: "Merge 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
Processing DSL script job_seed_standalone.groovy
Processing DSL script HadoopInputFormatJob.groovy
ERROR: startup failed:
HadoopInputFormatJob.groovy: 19: unable to resolve class CommonPropertiesFromDir
 @ line 19, column 1.
   import CommonPropertiesFromDir as commonProperties
   ^

1 error

Build failed in Jenkins: beam_SeedJob #2179

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b, 
no merge conflicts.
Setting status of 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2179/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision 8800c4ae431a6379b0c9dbad04e31b49a325fe14 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8800c4ae431a6379b0c9dbad04e31b49a325fe14
Commit message: "Merge 084ebd2dc33d148b91d6199c3faa7a53d80c7f2b into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
Processing DSL script job_seed_standalone.groovy
Processing DSL script HadoopInputFormatJob.groovy
ERROR: startup failed:
HadoopInputFormatJob.groovy: 19: unable to resolve class CommonPropertiesFromDir
 @ line 19, column 1.
   import CommonPropertiesFromDir as commonProperties
   ^

1 error

Build failed in Jenkins: beam_SeedJob #2178

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit 4671da6f82963aff97d160a6fa01e52dddb1ba78, 
no merge conflicts.
Setting status of 4671da6f82963aff97d160a6fa01e52dddb1ba78 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2178/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision 1e15fdf36c5b3bcaa78b8d122ed08574d0cb6855 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1e15fdf36c5b3bcaa78b8d122ed08574d0cb6855
Commit message: "Merge 4671da6f82963aff97d160a6fa01e52dddb1ba78 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
Processing DSL script job_seed_standalone.groovy
Processing DSL script HadoopInputFormatJob.groovy
ERROR: startup failed:
HadoopInputFormatJob.groovy: 19: unable to resolve class CommonProperties
 @ line 19, column 1.
   import CommonProperties as commonProperties
   ^

1 error

Jenkins build is back to normal : beam_SeedJob #2176

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_SeedJob #2175

[Apache Jenkins Server]

See 

--
GitHub pull request #5915 of commit aaa985e681a519b9dc6723d253769f7142c231ed, 
no merge conflicts.
Setting status of aaa985e681a519b9dc6723d253769f7142c231ed to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2175/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5915/*:refs/remotes/origin/pr/5915/*
 > git rev-parse refs/remotes/origin/pr/5915/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5915/merge^{commit} # timeout=10
Checking out Revision ec05e1b3d80e831dbdf0d430c7db7acc325e8574 
(refs/remotes/origin/pr/5915/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ec05e1b3d80e831dbdf0d430c7db7acc325e8574
Commit message: "Merge aaa985e681a519b9dc6723d253769f7142c231ed into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_HadoopInputFormat.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
ERROR: startup failed:
job_beam_PerformanceTests_Analysis.groovy: 19: unable to resolve class 
CommonProperites
 @ line 19, column 1.
   import CommonProperites as commonProperties
   ^

1 error

Build failed in Jenkins: beam_SeedJob #2174

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit a0969743a826820d58fc7558a44fce0d4ee859a0, 
no merge conflicts.
Setting status of a0969743a826820d58fc7558a44fce0d4ee859a0 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2174/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision a9d987f7c58c6dc4fbb14d0ff715fe1e4e476283 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a9d987f7c58c6dc4fbb14d0ff715fe1e4e476283
Commit message: "Merge a0969743a826820d58fc7558a44fce0d4ee859a0 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script NexmarkBigqueryProperties.groovy
org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast 
object 'org.codehaus.groovy.runtime.InvokerHelper$1@49ddee3b' with class 
'org.codehaus.groovy.runtime.InvokerHelper$1' to class 
'javaposse.jobdsl.dsl.JobParent'
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnSAM(DefaultTypeTransformation.java:405)
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnNumber(DefaultTypeTransformation.java:319)
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToType(DefaultTypeTransformation.java:232)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.castToType(ScriptBytecodeAdapter.java:603)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader.runScriptEngine(AbstractDslScriptLoader.groovy:103)
at sun.reflect.GeneratedMethodAccessor2812.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at 
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader$_runScripts_closure1.doCall(AbstractDslScriptLoader.groovy:59)
at sun.reflect.GeneratedMethodAccessor2810.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:430)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2040)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2025)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2078)
at org.codehaus.groovy.runtime.dgm$164.invoke(Unknown Source)
at 
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at 
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader.runScripts(AbstractDslScriptLoader.groovy:46)
at 

Build failed in Jenkins: beam_SeedJob #2173

[Apache Jenkins Server]

See 

--
GitHub pull request #5914 of commit 5f4a205fcffbb0b154406e5d95d56a32ca9bac70, 
no merge conflicts.
Setting status of 5f4a205fcffbb0b154406e5d95d56a32ca9bac70 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2173/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5914/*:refs/remotes/origin/pr/5914/*
 > git rev-parse refs/remotes/origin/pr/5914/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5914/merge^{commit} # timeout=10
Checking out Revision ce1d44336aa4e40320226ef1af2cdd00cd496133 
(refs/remotes/origin/pr/5914/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ce1d44336aa4e40320226ef1af2cdd00cd496133
Commit message: "Merge 5f4a205fcffbb0b154406e5d95d56a32ca9bac70 into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script NexmarkBigqueryProperties.groovy
org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast 
object 'org.codehaus.groovy.runtime.InvokerHelper$1@5f99ac32' with class 
'org.codehaus.groovy.runtime.InvokerHelper$1' to class 
'javaposse.jobdsl.dsl.JobParent'
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnSAM(DefaultTypeTransformation.java:405)
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.continueCastOnNumber(DefaultTypeTransformation.java:319)
at 
org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.castToType(DefaultTypeTransformation.java:232)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.castToType(ScriptBytecodeAdapter.java:603)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader.runScriptEngine(AbstractDslScriptLoader.groovy:103)
at sun.reflect.GeneratedMethodAccessor2812.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at 
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader$_runScripts_closure1.doCall(AbstractDslScriptLoader.groovy:59)
at sun.reflect.GeneratedMethodAccessor2810.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at 
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at groovy.lang.Closure.call(Closure.java:430)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2040)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2025)
at 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2078)
at org.codehaus.groovy.runtime.dgm$164.invoke(Unknown Source)
at 
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at 
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at 
javaposse.jobdsl.dsl.AbstractDslScriptLoader.runScripts(AbstractDslScriptLoader.groovy:46)
at 

Build failed in Jenkins: beam_SeedJob #2172

[Apache Jenkins Server]

See 

--
GitHub pull request #5913 of commit f68c0243c394e2c8fc808dec7f471f21b303c09b, 
no merge conflicts.
Setting status of f68c0243c394e2c8fc808dec7f471f21b303c09b to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2172/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5913/*:refs/remotes/origin/pr/5913/*
 > git rev-parse refs/remotes/origin/pr/5913/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5913/merge^{commit} # timeout=10
Checking out Revision 4a921723abf4d0446ee9a86e6205c0f7efe00ae6 
(refs/remotes/origin/pr/5913/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4a921723abf4d0446ee9a86e6205c0f7efe00ae6
Commit message: "Merge f68c0243c394e2c8fc808dec7f471f21b303c09b into 
6f6401d7b30f9095ba9bd581301be0dbae3e24fe"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
Processing DSL script job_PreCommit_Java.groovy
Processing DSL script job_PreCommit_Python.groovy
Processing DSL script job_PreCommit_Website_Merge.groovy
Processing DSL script job_PreCommit_Website_Stage.groovy
Processing DSL script job_PreCommit_Website_Test.groovy
Processing DSL script job_ReleaseCandidate_Python.groovy
Processing DSL script job_Release_Gradle_NightlySnapshot.groovy
Processing DSL script job_beam_PerformanceTests_Analysis.groovy
ERROR: startup failed:
job_beam_PerformanceTests_Analysis.groovy: 19: unable to resolve class 
CommonProperites
 @ line 19, column 1.
   import CommonProperites as commonProperties
   ^

1 error

Re: Performance issue in Beam 2.4 onwards

[Reuven Lax]

If we added something slow to the core library in order to better test
DirectRunner, that does sound like an unfortunate bug.

On Mon, Jul 9, 2018 at 11:21 PM Vojtech Janota 
wrote:

> Hi guys,
>
> Thank you for all of your feedback. I have created relevant issue in JIRA:
> https://issues.apache.org/jira/browse/BEAM-4750
>
> @Lukasz: me mentioning the DirectRunner was somewhat unfortunate - the
> bottleneck was introduced into the core library and so Flink and Spark
> runners would be impacted too
>
> Thanks,
> Vojta
>
> On Mon, Jul 9, 2018 at 5:48 PM, Lukasz Cwik  wrote:
>
>> Instead of reverting/working around specific checks/tests that the
>> DirectRunner is doing, have you considered using one of the other runners
>> like Flink or Spark with a local execution cluster. You won't hit the
>> validation/verification bottlenecks that DirectRunner specifically imposes.
>>
>> On Mon, Jul 9, 2018 at 8:46 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Thanks for the update Eugene.
>>>
>>> @Vojta: do you mind to create a Jira ? I will tackle a fix for that.
>>>
>>> Regards
>>> JB
>>>
>>> On 09/07/2018 17:33, Eugene Kirpichov wrote:
>>> > Hi -
>>> >
>>> > If I remember correctly, the reason for this change was to ensure that
>>> > the state is encodable at all. Prior to the change, there had been
>>> > situations where the coder specified on a state cell is buggy, absent
>>> or
>>> > set incorrectly (due to some issue in coder inference), but direct
>>> > runner did not detect this because it never tried to encode the state
>>> > cells - this would have blown up in any distributed runner.
>>> >
>>> > I think it should be possible to relax this and clone only values being
>>> > added to the state, rather than cloning the whole state on copy(). I
>>> > don't have time to work on this change myself, but I can review a PR if
>>> > someone else does.
>>> >
>>> > On Mon, Jul 9, 2018 at 8:28 AM Jean-Baptiste Onofré >> > > wrote:
>>> >
>>> > Hi Vojta,
>>> >
>>> > I fully agree, that's why it makes sense to wait Eugene's feedback.
>>> >
>>> > I remember we had some performance regression on the direct runner
>>> > identified thanks to Nexmark, but it has been addressed by
>>> reverting a
>>> > change.
>>> >
>>> > Good catch anyway !
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 09/07/2018 17:20, Vojtech Janota wrote:
>>> > > Hi Reuven,
>>> > >
>>> > > I'm not really complaining about DirectRunner. In fact it seems
>>> to
>>> > me as
>>> > > if what previously was considered as part of the "expensive extra
>>> > > checks" done by the DirectRunner is now done within the
>>> > > beam-runners-core-java library. Considering that all objects
>>> involved
>>> > > are immutable (in our case at least) and simple assignment is
>>> > > sufficient, the serialization-deserialization really seems as
>>> unwanted
>>> > > and hugely expensive correctness check. If there was a problem
>>> with
>>> > > identity copy, wasn't DirectRunner supposed to reveal it?
>>> > >
>>> > > Regards,
>>> > > Vojta
>>> > >
>>> > > On Mon, Jul 9, 2018 at 4:46 PM, Reuven Lax >> > 
>>> > > >> wrote:
>>> > >
>>> > > Hi Vojita,
>>> > >
>>> > > One problem is that the DirectRunner is designed for
>>> testing, not
>>> > > for performance. The DirectRunner currently does many
>>> > > purposely-inefficient things, the point of which is to better
>>> > expose
>>> > > potential bugs in tests. For example, the DirectRunner will
>>> > randomly
>>> > > shuffle the order of PCollections to ensure that your code
>>> > does not
>>> > > rely on ordering.  All of this adds cost, because the current
>>> > runner
>>> > > is designed for testing. There have been requests in the past
>>> > for an
>>> > > "optimized" local runner, however we don't currently have
>>> such
>>> > a thing.
>>> > >
>>> > > In this case, using coders to clone values is more correct.
>>> In a
>>> > > distributed environment using encode/decode is the only way
>>> to
>>> > copy
>>> > > values, and the DirectRunner is trying to ensure that your
>>> code is
>>> > > correct in a distributed environment.
>>> > >
>>> > > Reuven
>>> > >
>>> > > On Mon, Jul 9, 2018 at 7:22 AM Vojtech Janota
>>> > > mailto:vojta.jan...@gmail.com>
>>> > >>
>>> wrote:
>>> > >
>>> > > Hi,
>>> > >
>>> > > We are using Apache Beam in our project for some time
>>> now.
>>> > Since
>>> > > our datasets are of modest size, we have so far used
>>> > > DirectRunner as the computation easily fits onto a single
>>> > > machine. 

Re: CODEOWNERS for apache/beam repo

[Łukasz Gajowy]

+1. It will certainly be useful. I added myself (and a fellow contributor)
to some components (IO testing related mostly).

Thanks,
Łukasz

wt., 10 lip 2018 o 02:06 Udi Meiri  napisał(a):

> Hi everyone,
>
> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
> mechanism.
> Initial version is here: *https://github.com/apache/beam/pull/5909/files
> *
>
> I need help from the community in determining owners for each component.
> Feel free to directly edit the PR (if you have permission) or add a
> comment.
>
>
> Background
> The idea is to:
> 1. Document good review candidates for each component.
> 2. Help choose reviewers using the auto-assignment mechanism. The
> suggestion is in no way binding.
>
>
>

Re: Performance issue in Beam 2.4 onwards

[Vojtech Janota]

Hi guys,

Thank you for all of your feedback. I have created relevant issue in JIRA:
https://issues.apache.org/jira/browse/BEAM-4750

@Lukasz: me mentioning the DirectRunner was somewhat unfortunate - the
bottleneck was introduced into the core library and so Flink and Spark
runners would be impacted too

Thanks,
Vojta

On Mon, Jul 9, 2018 at 5:48 PM, Lukasz Cwik  wrote:

> Instead of reverting/working around specific checks/tests that the
> DirectRunner is doing, have you considered using one of the other runners
> like Flink or Spark with a local execution cluster. You won't hit the
> validation/verification bottlenecks that DirectRunner specifically imposes.
>
> On Mon, Jul 9, 2018 at 8:46 AM Jean-Baptiste Onofré 
> wrote:
>
>> Thanks for the update Eugene.
>>
>> @Vojta: do you mind to create a Jira ? I will tackle a fix for that.
>>
>> Regards
>> JB
>>
>> On 09/07/2018 17:33, Eugene Kirpichov wrote:
>> > Hi -
>> >
>> > If I remember correctly, the reason for this change was to ensure that
>> > the state is encodable at all. Prior to the change, there had been
>> > situations where the coder specified on a state cell is buggy, absent or
>> > set incorrectly (due to some issue in coder inference), but direct
>> > runner did not detect this because it never tried to encode the state
>> > cells - this would have blown up in any distributed runner.
>> >
>> > I think it should be possible to relax this and clone only values being
>> > added to the state, rather than cloning the whole state on copy(). I
>> > don't have time to work on this change myself, but I can review a PR if
>> > someone else does.
>> >
>> > On Mon, Jul 9, 2018 at 8:28 AM Jean-Baptiste Onofré > > > wrote:
>> >
>> > Hi Vojta,
>> >
>> > I fully agree, that's why it makes sense to wait Eugene's feedback.
>> >
>> > I remember we had some performance regression on the direct runner
>> > identified thanks to Nexmark, but it has been addressed by
>> reverting a
>> > change.
>> >
>> > Good catch anyway !
>> >
>> > Regards
>> > JB
>> >
>> > On 09/07/2018 17:20, Vojtech Janota wrote:
>> > > Hi Reuven,
>> > >
>> > > I'm not really complaining about DirectRunner. In fact it seems to
>> > me as
>> > > if what previously was considered as part of the "expensive extra
>> > > checks" done by the DirectRunner is now done within the
>> > > beam-runners-core-java library. Considering that all objects
>> involved
>> > > are immutable (in our case at least) and simple assignment is
>> > > sufficient, the serialization-deserialization really seems as
>> unwanted
>> > > and hugely expensive correctness check. If there was a problem
>> with
>> > > identity copy, wasn't DirectRunner supposed to reveal it?
>> > >
>> > > Regards,
>> > > Vojta
>> > >
>> > > On Mon, Jul 9, 2018 at 4:46 PM, Reuven Lax > > 
>> > > >> wrote:
>> > >
>> > > Hi Vojita,
>> > >
>> > > One problem is that the DirectRunner is designed for testing,
>> not
>> > > for performance. The DirectRunner currently does many
>> > > purposely-inefficient things, the point of which is to better
>> > expose
>> > > potential bugs in tests. For example, the DirectRunner will
>> > randomly
>> > > shuffle the order of PCollections to ensure that your code
>> > does not
>> > > rely on ordering.  All of this adds cost, because the current
>> > runner
>> > > is designed for testing. There have been requests in the past
>> > for an
>> > > "optimized" local runner, however we don't currently have such
>> > a thing.
>> > >
>> > > In this case, using coders to clone values is more correct.
>> In a
>> > > distributed environment using encode/decode is the only way to
>> > copy
>> > > values, and the DirectRunner is trying to ensure that your
>> code is
>> > > correct in a distributed environment.
>> > >
>> > > Reuven
>> > >
>> > > On Mon, Jul 9, 2018 at 7:22 AM Vojtech Janota
>> > > mailto:vojta.jan...@gmail.com>
>> > >>
>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > We are using Apache Beam in our project for some time now.
>> > Since
>> > > our datasets are of modest size, we have so far used
>> > > DirectRunner as the computation easily fits onto a single
>> > > machine. Recently we upgraded Beam from 2.2 to 2.4 and
>> > found out
>> > > that performance of our pipelines drastically
>> deteriorated.
>> > > Pipelines that took ~3 minutes with 2.2 do not finish
>> within
>> > > hours now. We tried to isolate the change that causes the
>> > > slowdown and