date:20170917

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3983

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 8.44 MB...]
[INFO] 2017-09-17T07:26:19.242Z: (4c4c43e9a10be65): Executing operation 
PAssert$311/GroupGlobally/GroupDummyAndContents/Read+PAssert$311/GroupGlobally/GroupDummyAndContents/GroupByWindow+PAssert$311/GroupGlobally/Values/Values/Map+PAssert$311/GroupGlobally/ParDo(Concat)+PAssert$311/GetPane/Map+PAssert$311/RunChecks+PAssert$311/VerifyAssertions/ParDo(DefaultConclude)
[INFO] 2017-09-17T07:26:21.041Z: (b03841bd8caa): Executing operation 
Sum/Combine.perKey(SumInts)/GroupByKey/Close
[INFO] 2017-09-17T07:26:21.058Z: (b03841bd8caa6abb): Executing operation 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Create
[INFO] 2017-09-17T07:26:21.069Z: (48b872ba2d90a6fa): Executing operation 
PAssert$318/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2017-09-17T07:26:21.117Z: (48b872ba2d90ab4e): Executing operation 
Sum/Combine.perKey(SumInts)/GroupByKey/Read+Sum/Combine.perKey(SumInts)/Combine.GroupedValues+Sum/Combine.perKey(SumInts)/Combine.GroupedValues/Extract+Sum/Values/Values/Map+PAssert$318/GroupGlobally/Window.Into()/Window.Assign+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map+PAssert$318/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows)+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify+View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write+PAssert$318/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$318/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$318/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$318/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] 2017-09-17T07:26:23.093Z: (1b48c1db7fd0e0f1): Cleaning up.
[INFO] 2017-09-17T07:26:23.098Z: (1b48c1db7fd0e255): Stopping worker pool...
[INFO] 2017-09-17T07:26:25.404Z: (fcdfdb77db1c5fba): Executing operation 
Combine.perKey(UniqueInts)/GroupByKey/Close
[INFO] 2017-09-17T07:26:25.425Z: (e9e03cb7bcd0152b): Executing operation 
Combine.globally(UniqueInts)/Combine.perKey(UniqueInts)/GroupByKey/Close
[INFO] 2017-09-17T07:26:25.456Z: (fcdfdb77db1c5d32): Executing operation 
PAssert$326/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2017-09-17T07:26:25.485Z: (e9e03cb7bcd013a3): Executing operation 
PAssert$325/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2017-09-17T07:26:25.570Z: (e9e03cb7bcd01b9e): Executing operation 
Combine.perKey(UniqueInts)/GroupByKey/Read+Combine.perKey(UniqueInts)/Combine.GroupedValues+Combine.perKey(UniqueInts)/Combine.GroupedValues/Extract+PAssert$326/GroupGlobally/Window.Into()/Window.Assign+PAssert$326/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows)+PAssert$326/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$326/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$326/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$326/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] 2017-09-17T07:26:25.593Z: (fcdfdb77db1c56de): Executing operation 
Combine.globally(UniqueInts)/Combine.perKey(UniqueInts)/GroupByKey/Read+Combine.globally(UniqueInts)/Combine.perKey(UniqueInts)/Combine.GroupedValues+Combine.globally(UniqueInts)/Combine.perKey(UniqueInts)/Combine.GroupedValues/Extract+Combine.globally(UniqueInts)/Values/Values/Map+PAssert$325/GroupGlobally/Window.Into()/Window.Assign+PAssert$325/GroupGlobally/GatherAllOutputs/ParDo(ReifyTimestampsAndWindows)+Combine.globally(UniqueInts)/View.AsIterable/View.CreatePCollectionView/ParDo(ToIsmRecordForGlobalWindow)+PAssert$325/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map+PAssert$325/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign+PAssert$325/GroupGlobally/GatherAllOutputs/GroupByKey/Reify+PAssert$325/GroupGlobally/GatherAllOutputs/GroupByKey/Write
[INFO] 2017-09-17T07:26:24.944Z: (8914df7ab518299f): Executing operation 
PAssert$321/GroupGlobally/GatherAllOutputs/GroupByKey/Close
[INFO] 2017-09-17T07:26:24.967Z: (6cce6381907c922a): Executing operation 
PAssert$321/GroupGlobally/GroupDummyAndContents/Create
[INFO] 2017-09-17T07:26:25.033Z: (8914df7ab5182f97): Executing operation 
PAssert$321/GroupGlobally/Create.Values/Read(CreateSource)+PAssert$321/GroupGlobally/WindowIntoDummy/Window.Assign+PAssert$321/GroupGlobally/GroupDummyAndContents/Reify+PAssert$321/GroupGlobally/GroupDummyAndContents/Write
[INFO] 2017-09-17T07:26:25.058Z: (6cce6381907c953e): Executing operation 
PAssert$321/GroupGl

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4809

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 4.06 MB...]
2017-09-17T07:31:17.446 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-javadoc ---
2017-09-17T07:31:17.554 [INFO] 
2017-09-17T07:31:17.554 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-javadoc ---
2017-09-17T07:31:17.661 [INFO] 
2017-09-17T07:31:17.661 [INFO] --- 
reproducible-build-maven-plugin:0.3:strip-jar (default) @ 
beam-sdks-java-javadoc ---
2017-09-17T07:31:17.662 [INFO] Stripping 

2017-09-17T07:31:17.806 [INFO] 
2017-09-17T07:31:17.806 [INFO] --- maven-dependency-plugin:3.0.1:analyze-only 
(default) @ beam-sdks-java-javadoc ---
2017-09-17T07:31:17.807 [INFO] Skipping plugin execution
2017-09-17T07:31:17.914 [INFO] 
2017-09-17T07:31:17.914 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-sdks-java-javadoc ---
2017-09-17T07:31:17.915 [INFO] Installing 

 to 

2017-09-17T07:31:17.916 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-09-17T07:31:19.366 [INFO]  
   
2017-09-17T07:31:19.366 [INFO] 

2017-09-17T07:31:19.366 [INFO] Building Apache Beam :: Parent 2.2.0-SNAPSHOT
2017-09-17T07:31:19.366 [INFO] 

2017-09-17T07:31:19.388 [WARNING] The POM for 
org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency 
information available
2017-09-17T07:31:19.390 [WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
2017-09-17T07:31:19.391 [INFO] 
2017-09-17T07:31:19.391 [INFO] --- coveralls-maven-plugin:4.3.0:report 
(default-cli) @ beam-parent ---
2017-09-17T07:31:19.421 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.pom
2017-09-17T07:31:19.448 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.pom
 (5 KB at 173.0 KB/sec)
2017-09-17T07:31:19.451 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.8.3/jackson-core-2.8.3.pom
2017-09-17T07:31:19.477 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.8.3/jackson-core-2.8.3.pom
 (6 KB at 203.4 KB/sec)
2017-09-17T07:31:19.481 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.3/jackson-annotations-2.8.3.pom
2017-09-17T07:31:19.507 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.3/jackson-annotations-2.8.3.pom
 (2 KB at 69.4 KB/sec)
2017-09-17T07:31:19.509 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.8.3/jackson-databind-2.8.3.pom
2017-09-17T07:31:19.549 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.8.3/jackson-databind-2.8.3.pom
 (6 KB at 132.5 KB/sec)
2017-09-17T07:31:19.552 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.5.0.201609210915-r/org.eclipse.jgit-4.5.0.201609210915-r.pom
2017-09-17T07:31:19.578 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.5.0.201609210915-r/org.eclipse.jgit-4.5.0.201609210915-r.pom
 (10 KB at 356.9 KB/sec)
2017-09-17T07:31:19.579 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/4.5.0.201609210915-r/org.eclipse.jgit-parent-4.5.0.201609210915-r.pom
2017-09-17T07:31:19.606 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/4.5.0.201609210915-r/org.eclipse.jgit-parent-4.5.0.201609210915-

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3150

2017-09-17 Thread Apache Jenkins Server

See

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #3087

2017-09-17 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #3984

2017-09-17 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4810

2017-09-17 Thread Apache Jenkins Server

See

[jira] [Updated] (BEAM-2775) HadoopInputFormatIOTest fails with NoSuchMethodError

2017-09-17 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated BEAM-2775:
-
Description: 
As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:

{code}
[ERROR] 
testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
  Time elapsed: 0.007 s  <<< ERROR!
java.lang.NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodsAreEitherBeanMethodOrKnownMethod(PipelineOptionsFactory.java:1255)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:990)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:616)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:550)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:569)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:525)
  at 
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
  at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
{code}
Here is related code:
{code}
checkArgument(unknownMethods.isEmpty(),
"Methods %s on [%s] do not conform to being bean properties.",

FluentIterable.from(unknownMethods).transform(ReflectHelpers.METHOD_FORMATTER),
iface.getName());
{code}
This led to test failure of subtests:
{code}
[ERROR] 
testReadersGetFractionConsumed(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
  Time elapsed: 0.001 s  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.sdk.options.PipelineOptionsFactory
  at 
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
  at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
  at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest.(HadoopInputFormatIOTest.java:69)
{code}

  was:
As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
{code}
[ERROR] 
testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
  Time elapsed: 0.007 s  <<< ERROR!
java.lang.NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodsAreEitherBeanMethodOrKnownMethod(PipelineOptionsFactory.java:1255)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:990)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:616)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:550)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:569)
  at 
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:525)
  at 
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
  at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
{code}
Here is related code:
{code}
checkArgument(unknownMethods.isEmpty(),
"Methods %s on [%s] do not conform to being bean properties.",

FluentIterable.from(unknownMethods).transform(ReflectHelpers.METHOD_FORMATTER),
iface.getName());
{code}
This led to test failure of subtests:
{code}
[ERROR] 
testReadersGetFractionConsumed(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
  Time elapsed: 0.001 s  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.beam.sdk.options.PipelineOptionsFactory
  at 
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:404)
  at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:262)
  at 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest.(HadoopInputFormatIOTest.java:69)
{code}


> HadoopInputFormatIOTest fails with NoSuchMethodError
> 
>
> Key: BEAM-2775
> URL: https://issues.apache.org/jira/browse/BEAM-2775
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Luke Cwik
>
> As of commit 8503adbbc3a590cd0dc2939f6a45d335682a9442, I got:
> {code}
> [ERROR] 
> testReadObjectCreationWithConfigurationValueTranslation(org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOTest)
>   Time elapsed: 0.007 s  <<< ERROR!
> java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.

[jira] [Updated] (BEAM-2805) Potential arithmetic overflow in Generator#nextAuctionLengthMs()

2017-09-17 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated BEAM-2805:
-
Description: 
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions * 
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}
The multiplication is done on 32-bit integers while long is expected 
(numEventsForAuctions).

There is possibility for arithmetic overflow

  was:
{code}
long numEventsForAuctions =
(config.configuration.numInFlightAuctions * 
GeneratorConfig.PROPORTION_DENOMINATOR)
/ GeneratorConfig.AUCTION_PROPORTION;
{code}

The multiplication is done on 32-bit integers while long is expected 
(numEventsForAuctions).

There is possibility for arithmetic overflow


> Potential arithmetic overflow in Generator#nextAuctionLengthMs()
> 
>
> Key: BEAM-2805
> URL: https://issues.apache.org/jira/browse/BEAM-2805
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Reuven Lax
>Priority: Minor
>
> {code}
> long numEventsForAuctions =
> (config.configuration.numInFlightAuctions * 
> GeneratorConfig.PROPORTION_DENOMINATOR)
> / GeneratorConfig.AUCTION_PROPORTION;
> {code}
> The multiplication is done on 32-bit integers while long is expected 
> (numEventsForAuctions).
> There is possibility for arithmetic overflow



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-1962) Connection should be closed in case start() throws exception

2017-09-17 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated BEAM-1962:
-
Description: 
In JmsIO#start() :
{code}
  try {
Connection connection;
if (spec.getUsername() != null) {
  connection =
  connectionFactory.createConnection(spec.getUsername(), 
spec.getPassword());
} else {
  connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
  } catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
  }
{code}
If start() throws exception, connection should be closed.

  was:
In JmsIO#start() :

{code}
  try {
Connection connection;
if (spec.getUsername() != null) {
  connection =
  connectionFactory.createConnection(spec.getUsername(), 
spec.getPassword());
} else {
  connection = connectionFactory.createConnection();
}
connection.start();
this.connection = connection;
  } catch (Exception e) {
throw new IOException("Error connecting to JMS", e);
  }
{code}
If start() throws exception, connection should be closed.


> Connection should be closed in case start() throws exception
> 
>
> Key: BEAM-1962
> URL: https://issues.apache.org/jira/browse/BEAM-1962
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ted Yu
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> In JmsIO#start() :
> {code}
>   try {
> Connection connection;
> if (spec.getUsername() != null) {
>   connection =
>   connectionFactory.createConnection(spec.getUsername(), 
> spec.getPassword());
> } else {
>   connection = connectionFactory.createConnection();
> }
> connection.start();
> this.connection = connection;
>   } catch (Exception e) {
> throw new IOException("Error connecting to JMS", e);
>   }
> {code}
> If start() throws exception, connection should be closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3151

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 43.99 KB...]
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.2.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.2.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.2.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.2.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-z40wYh-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpqq6Hb9pip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-z40wYh-build/setup.py", line 198, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-z40wYh-build/setup.py", line 138, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, 
mock, pyasn1, pyasn1-modules, rsa, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: protobuf 3.4

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #3088

2017-09-17 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4811

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 3.63 MB...]
from apitools.base.protorpclite import message_types
  File 
"
 line 25, in 
from apitools.base.protorpclite import messages
  File 
"
 line 1165, in 
class Field(six.with_metaclass(_FieldMeta, object)):
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) 
subclass of the metaclasses of all its bases
ERROR: InvocationError: 
'
 apache_beam/examples/complete/autocomplete_test.py'
___ summary 
ERROR:   docs: commands failed
  lint: commands succeeded
  py27: commands succeeded
  py27cython: commands succeeded
ERROR:   py27gcp: commands failed
2017-09-17T19:21:45.034 [ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit 
value: 1)
at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at 
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at jenkins.maven3.agent.Maven33Main.launch(Maven33Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:139)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[JENKINS] Archiving disabled
2017-09-17T19:21:46.003 [INFO]  
   
2017-09-17T19:21:46.003 [INFO] 

2017-09-17T19:21:46.003 [INFO] Skipping Apache Beam :: Parent
2017-09-17T19:2

Build failed in Jenkins: beam_PostCommit_Python_Verify #3152

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 43.99 KB...]
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.2.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.2.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.2.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.2.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-YIC9bl-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpDu7NuLpip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-YIC9bl-build/setup.py", line 198, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-YIC9bl-build/setup.py", line 138, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, pbr, funcsigs, 
mock, pyasn1, pyasn1-modules, rsa, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: protobuf 3.4

[jira] [Created] (BEAM-2963) Propagate pipeline protos through Dataflow API from Java

2017-09-17 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-2963:
-

 Summary: Propagate pipeline protos through Dataflow API from Java
 Key: BEAM-2963
 URL: https://issues.apache.org/jira/browse/BEAM-2963
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The Java-specific blobs transmitted to Dataflow need more context, in the form 
of portability framework protos.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3858: [BEAM-2884] Move DoFnInfo to SDK util

2017-09-17 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3858

[BEAM-2884] Move DoFnInfo to SDK util

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a DoFn without much context. The Java SDK harness has the same need, and relies 
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness 
understands.

This also helps to break the dependency between the Java SDK harness and 
the Dataflow runner codebase, by just moving it to be an allowed dependency.

This will require a new worker image.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DoFnInfo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3858.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3858


commit d8f03d7a3d749dcd279b35377e25dc8884dc5a64
Author: Kenneth Knowles 
Date:   2017-09-16T22:16:56Z

Move DoFnInfo to SDK util

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a
DoFn without much context. The Java SDK harness has the same need, and 
relies
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness
understands.




---

[jira] [Commented] (BEAM-2884) Dataflow runs portable pipelines

2017-09-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169446#comment-16169446
 ] 

ASF GitHub Bot commented on BEAM-2884:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3858

[BEAM-2884] Move DoFnInfo to SDK util

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a DoFn without much context. The Java SDK harness has the same need, and relies 
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness 
understands.

This also helps to break the dependency between the Java SDK harness and 
the Dataflow runner codebase, by just moving it to be an allowed dependency.

This will require a new worker image.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DoFnInfo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3858.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3858


commit d8f03d7a3d749dcd279b35377e25dc8884dc5a64
Author: Kenneth Knowles 
Date:   2017-09-16T22:16:56Z

Move DoFnInfo to SDK util

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a
DoFn without much context. The Java SDK harness has the same need, and 
relies
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness
understands.




> Dataflow runs portable pipelines
> 
>
> Key: BEAM-2884
> URL: https://issues.apache.org/jira/browse/BEAM-2884
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>  Labels: portability
>
> Dataflow should run pipelines using the full portability API as currently 
> defined:
> https://s.apache.org/beam-fn-api 
> https://s.apache.org/beam-runner-api
> https://s.apache.org/beam-job-api
> https://s.apache.org/beam-fn-api-container-contract
> This issue tracks its adoption of the portability framework. New Fn API and 
> other features will be tracked separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3859: [BEAM-2884] Send portable protos for ParDo in Dataf...

2017-09-17 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3859

[BEAM-2884] Send portable protos for ParDo in DataflowRunner

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Instead of just sending Dataflow a Java-serialized `DoFnInfo`, send it as 
part of a portable payload of the form: `PTransform` proto > `ParDoPayload` > 
`SdkFunctionSpec` > `FunctionSpec`

A couple notes:

 - This is build on #3858 which is trivial. We could possibly do a worker 
dance for both together, but it is nice for them to be separate commits.
 - The full `DoFnInfo` is still sent, as the Java SDK harness depends on 
many of its details and will continue to do so for the foreseeable future. For 
example, the default "main" output, which is not part of the portable model. 
The contents of `DoFnInfo` can be refined separately.
 - There are pieces left blank, such as the environment of the 
`SdkFunctionSpec`.
 - It may also be possible that Dataflow doesn't need quite so much context 
in the protos. It only needs the pieces that the runner harness uses to 
construct an instruction graph for the harness.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ParDoPayload

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3859


commit d8f03d7a3d749dcd279b35377e25dc8884dc5a64
Author: Kenneth Knowles 
Date:   2017-09-16T22:16:56Z

Move DoFnInfo to SDK util

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a
DoFn without much context. The Java SDK harness has the same need, and 
relies
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness
understands.

commit 478adfab4d94cf14826a107766b50a2c31ca5cc1
Author: Kenneth Knowles 
Date:   2017-09-16T22:26:49Z

Send ParDoPayload to Dataflow instead of just DoFnInfo




---

[jira] [Commented] (BEAM-2884) Dataflow runs portable pipelines

2017-09-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169453#comment-16169453
 ] 

ASF GitHub Bot commented on BEAM-2884:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3859

[BEAM-2884] Send portable protos for ParDo in DataflowRunner

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

Instead of just sending Dataflow a Java-serialized `DoFnInfo`, send it as 
part of a portable payload of the form: `PTransform` proto > `ParDoPayload` > 
`SdkFunctionSpec` > `FunctionSpec`

A couple notes:

 - This is build on #3858 which is trivial. We could possibly do a worker 
dance for both together, but it is nice for them to be separate commits.
 - The full `DoFnInfo` is still sent, as the Java SDK harness depends on 
many of its details and will continue to do so for the foreseeable future. For 
example, the default "main" output, which is not part of the portable model. 
The contents of `DoFnInfo` can be refined separately.
 - There are pieces left blank, such as the environment of the 
`SdkFunctionSpec`.
 - It may also be possible that Dataflow doesn't need quite so much context 
in the protos. It only needs the pieces that the runner harness uses to 
construct an instruction graph for the harness.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam ParDoPayload

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3859


commit d8f03d7a3d749dcd279b35377e25dc8884dc5a64
Author: Kenneth Knowles 
Date:   2017-09-16T22:16:56Z

Move DoFnInfo to SDK util

Previously, the DoFnInfo wrapped things just enough for Dataflow to execute 
a
DoFn without much context. The Java SDK harness has the same need, and 
relies
on DoFnInfo. Effectively, DoFnInfo is the UDF that the Java SDK harness
understands.

commit 478adfab4d94cf14826a107766b50a2c31ca5cc1
Author: Kenneth Knowles 
Date:   2017-09-16T22:26:49Z

Send ParDoPayload to Dataflow instead of just DoFnInfo




> Dataflow runs portable pipelines
> 
>
> Key: BEAM-2884
> URL: https://issues.apache.org/jira/browse/BEAM-2884
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>  Labels: portability
>
> Dataflow should run pipelines using the full portability API as currently 
> defined:
> https://s.apache.org/beam-fn-api 
> https://s.apache.org/beam-runner-api
> https://s.apache.org/beam-job-api
> https://s.apache.org/beam-fn-api-container-contract
> This issue tracks its adoption of the portability framework. New Fn API and 
> other features will be tracked separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] 02/02: This closes #319

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 0d96bcc781fddaf3c5d8407a85523687c7bc2263
Merge: 4a15dcf 8b94fc1
Author: Mergebot 
AuthorDate: Sun Sep 17 23:22:49 2017 +

This closes #319

 src/get-started/quickstart-java.md | 4 +++-
 src/get-started/quickstart-py.md   | 4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/02: [BEAM-2510] Link to SDK pages from the quickstarts

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8b94fc124e6c2ba953ea3c7810435e0414b4f517
Author: melissa 
AuthorDate: Fri Sep 8 18:11:11 2017 -0700

[BEAM-2510] Link to SDK pages from the quickstarts
---
 src/get-started/quickstart-java.md | 4 +++-
 src/get-started/quickstart-py.md   | 4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index 8b9a5f6..3c2fb82 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -228,7 +228,9 @@ barrenly: 1
 
 ## Next Steps
 
-* Learn more about these WordCount examples in the [WordCount Example 
Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
+* Learn more about the [Beam SDK for Java]({{ site.baseurl 
}}/documentation/sdks/java/)
+  and look through the [Java SDK API reference]({{ site.baseurl 
}}/documentation/sdks/javadoc).
+* Walk through these WordCount examples in the [WordCount Example 
Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
 * Dive in to some of our favorite [articles and presentations]({{ site.baseurl 
}}/documentation/resources).
 * Join the Beam [users@]({{ site.baseurl }}/get-started/support#mailing-lists) 
mailing list.
 
diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index f5cf2aa..cae9db9 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -118,7 +118,9 @@ python -m apache_beam.examples.wordcount --input 
gs://dataflow-samples/shakespea
 
 ## Next Steps
 
-* Learn more about these WordCount examples in the [WordCount Example 
Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
+* Learn more about the [Beam SDK for Python]({{ site.baseurl 
}}/documentation/sdks/python/)
+  and look through the [Python SDK API reference]({{ site.baseurl 
}}/documentation/sdks/pydoc).
+* Walk through these WordCount examples in the [WordCount Example 
Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
 * Dive in to some of our favorite [articles and presentations]({{ site.baseurl 
}}/documentation/resources).
 * Join the Beam [users@]({{ site.baseurl }}/get-started/support#mailing-lists) 
mailing list.
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (3a6d0b1 -> 0d96bcc)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 3a6d0b1  This closes #318
 discard 620e761  [BEAM-2869] Add dates to releases on downloads page
 new 8b94fc1  [BEAM-2510] Link to SDK pages from the quickstarts
 new 0d96bcc  This closes #319

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3a6d0b1)
\
 N -- N -- N   refs/heads/mergebot (0d96bcc)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/get-started/downloads.md   | 16 
 src/get-started/quickstart-java.md |  4 +++-
 src/get-started/quickstart-py.md   |  4 +++-
 3 files changed, 14 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit bf692c6f2d8751b94d4033ce83af0f4af502dbae
Author: Mergebot 
AuthorDate: Sun Sep 17 23:25:39 2017 +

Prepare repository for deployment.
---
 content/get-started/quickstart-java/index.html | 4 +++-
 content/get-started/quickstart-py/index.html   | 4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/content/get-started/quickstart-java/index.html 
b/content/get-started/quickstart-java/index.html
index dd2e23e..3808dd6 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -355,7 +355,9 @@ barrenly: 1
 Next Steps
 
 
-  Learn more about these WordCount examples in the WordCount Example Walkthrough.
+  Learn more about the Beam SDK for 
Java
+and look through the Java SDK API 
reference.
+  Walk through these WordCount examples in the WordCount Example Walkthrough.
   Dive in to some of our favorite articles and presentations.
   Join the Beam users@ 
mailing list.
 
diff --git a/content/get-started/quickstart-py/index.html 
b/content/get-started/quickstart-py/index.html
index 1a467ef..2e58f58 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -285,7 +285,9 @@ python -m apache_beam.examples.wordcount --input 
gs://dataflow-samples/shakespea
 Next Steps
 
 
-  Learn more about these WordCount examples in the WordCount Example Walkthrough.
+  Learn more about the Beam SDK for 
Python
+and look through the Python SDK API 
reference.
+  Walk through these WordCount examples in the WordCount Example Walkthrough.
   Dive in to some of our favorite articles and presentations.
   Join the Beam users@ 
mailing list.
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (0d96bcc -> fa0cc80)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 0d96bcc  This closes #319
 add bf692c6  Prepare repository for deployment.
 new e37e447  [BEAM-2869] Add dates to releases on downloads page
 new fa0cc80  This closes #318

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html |  4 +++-
 content/get-started/quickstart-py/index.html   |  4 +++-
 src/get-started/downloads.md   | 16 
 3 files changed, 14 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] branch asf-site updated (4a15dcf -> bf692c6)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4a15dcf  Prepare repository for deployment.
 add 8b94fc1  [BEAM-2510] Link to SDK pages from the quickstarts
 add 0d96bcc  This closes #319
 new bf692c6  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-java/index.html | 4 +++-
 content/get-started/quickstart-py/index.html   | 4 +++-
 src/get-started/quickstart-java.md | 4 +++-
 src/get-started/quickstart-py.md   | 4 +++-
 4 files changed, 12 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 02/02: This closes #318

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit fa0cc8028a31f61fe4665ee0622b2e6b947546a8
Merge: bf692c6 e37e447
Author: Mergebot 
AuthorDate: Sun Sep 17 23:26:05 2017 +

This closes #318

 src/get-started/downloads.md | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/02: [BEAM-2869] Add dates to releases on downloads page

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e37e447b807ab103d61d754d5b82f676e3ddcb7d
Author: melissa 
AuthorDate: Fri Sep 8 14:23:53 2017 -0700

[BEAM-2869] Add dates to releases on downloads page
---
 src/get-started/downloads.md | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/get-started/downloads.md b/src/get-started/downloads.md
index a4d575d..b50da91 100644
--- a/src/get-started/downloads.md
+++ b/src/get-started/downloads.md
@@ -54,42 +54,42 @@ Additionally, any API may change before the first stable 
release, i.e., between
 
 ## Releases
 
-### 2.1.0
+### 2.1.0 (2017-08-23)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/2.1.0/apache-beam-2.1.0-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12340528).
 
-### 2.0.0
+### 2.0.0 (2017-05-17)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/2.0.0/apache-beam-2.0.0-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12339746).
 
-### 0.6.0
+### 0.6.0 (2017-03-11)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.6.0/apache-beam-0.6.0-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12339256).
 
-### 0.5.0
+### 0.5.0 (2017-02-02)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.5.0/apache-beam-0.5.0-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338859).
 
-### 0.4.0
+### 0.4.0 (2016-12-29)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.4.0/apache-beam-0.4.0-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338590).
 
-### 0.3.0-incubating
+### 0.3.0-incubating (2016-10-31)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338051).
 
-### 0.2.0-incubating
+### 0.2.0-incubating (2016-08-08)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zip&action=download).
 
 [Release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12335766).
 
-### 0.1.0-incubating
+### 0.1.0-incubating (2016-06-15)
 Official [source code 
download](https://www.apache.org/dyn/closer.cgi?filename=beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zip&action=download).
 
 The first incubating release of Apache Beam.

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/01: Prepare repository for deployment.

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 0ad1d91c0d00ce67335a2afb344162f38a65a647
Author: Mergebot 
AuthorDate: Sun Sep 17 23:28:08 2017 +

Prepare repository for deployment.
---
 content/get-started/downloads/index.html | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/content/get-started/downloads/index.html 
b/content/get-started/downloads/index.html
index f95fffa..6b1a464 100644
--- a/content/get-started/downloads/index.html
+++ b/content/get-started/downloads/index.html
@@ -194,42 +194,42 @@ may change at any point and are not guaranteed to remain 
compatible across versi
 
 Releases
 
-2.1.0
+2.1.0 (2017-08-23)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/2.1.0/apache-beam-2.1.0-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12340528";>Release
 notes.
 
-2.0.0
+2.0.0 (2017-05-17)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/2.0.0/apache-beam-2.0.0-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12339746";>Release
 notes.
 
-0.6.0
+0.6.0 (2017-03-11)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.6.0/apache-beam-0.6.0-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12339256";>Release
 notes.
 
-0.5.0
+0.5.0 (2017-02-02)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.5.0/apache-beam-0.5.0-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338859";>Release
 notes.
 
-0.4.0
+0.4.0 (2016-12-29)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.4.0/apache-beam-0.4.0-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338590";>Release
 notes.
 
-0.3.0-incubating
+0.3.0-incubating (2016-10-31)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12338051";>Release
 notes.
 
-0.2.0-incubating
+0.2.0-incubating (2016-08-08)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zip&action=download";>source
 code download.
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12335766";>Release
 notes.
 
-0.1.0-incubating
+0.1.0-incubating (2016-06-15)
 Official https://www.apache.org/dyn/closer.cgi?filename=beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zip&action=download";>source
 code download.
 
 The first incubating release of Apache Beam.

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (bf692c6 -> 0ad1d91)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from bf692c6  Prepare repository for deployment.
 add e37e447  [BEAM-2869] Add dates to releases on downloads page
 add fa0cc80  This closes #318
 new 0ad1d91  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/downloads/index.html | 16 
 src/get-started/downloads.md | 16 
 2 files changed, 16 insertions(+), 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: [BEAM-1534] Add docker images documentation to the website

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d3f3543f2adceb6be7250068d022061eb7506365
Author: Ismaël Mejía 
AuthorDate: Tue Aug 22 10:59:27 2017 +0200

[BEAM-1534] Add docker images documentation to the website
---
 src/_includes/header.html   |   1 +
 src/contribute/docker-images.md | 183 
 2 files changed, 184 insertions(+)

diff --git a/src/_includes/header.html b/src/_includes/header.html
index 98e7899..839b4ea 100644
--- a/src/_includes/header.html
+++ b/src/_includes/header.html
@@ -85,6 +85,7 @@
 Design Principles
 Ongoing Projects
 Source Repository
+Docker 
Images
 
 Promotion
 Presentation Materials
diff --git a/src/contribute/docker-images.md b/src/contribute/docker-images.md
new file mode 100644
index 000..0cb19da
--- /dev/null
+++ b/src/contribute/docker-images.md
@@ -0,0 +1,183 @@
+---
+layout: default
+title: 'Beam Docker Images'
+permalink: /contribute/docker-images/
+---
+
+# Docker Images
+
+Docker images allow to create a reproducible environment to build and test
+Beam. You can use the docker images by using the provided [Docker 
scripts](https://github.com/apache/beam/tree/master/sdks/java/build-tools/src/main/resources/docker).
+
+In this directory you will find scripts to build and run docker images for
+different purposes:
+
+- [file](#file): Create a Docker container from a Beam source code .zip file
+  in a given environment. It is useful to test a specific version of Beam,
+  for example to validate a release vote.
+
+- [git](#git): Same as file but the Beam source code comes from the git 
repository,
+  you can choose a given branch/tag/pull-request. Useful to test in a specific
+  environment.
+
+- [release](#release): It builds an end-user distribution of the latest 
version of Beam
+  and its dependencies. Useful for end-users who want to have a ready to use
+  container with Beam (Python only for the moment).
+
+## File based image
+
+If you want to build a container with a ready JDK 8 environment to test Beam:
+
+```
+cd file/openjdk8
+docker build -t beam:openjdk8 .
+```
+
+If you want to build a container with JDK 7:
+
+```
+cd file/openjdk7
+docker build -t beam:openjdk7 .
+```
+
+When you run the image it downloads the specific version of Beam given the
+environment variables. By default it downloads the source code of the latest
+master. If you want to download the latest master and execute the tests:
+
+```
+docker run -it beam:openjdk8 mvn clean verify -Prelease
+```
+
+If you want to have an interactive session you can run a bash prompt:
+
+```
+docker run -it beam:openjdk8 /bin/bash
+```
+
+Inside the container you can test a specific module by running Maven. You
+have to change MODULE_PATH for the given module path. For example to test
+HBaseIO you should write its path `sdks/java/io/hbase` in the place of
+MODULE_PATH:
+
+```
+mvn --projects MODULE_PATH clean verify -Prelease
+```
+
+### Configuring the runtime via the environment variables
+
+You can run different versions of Beam by passing the specific environment
+variables:
+
+`URL`: The URL with the file containing the specific source code.  
+`SRC_FILE`: The downloaded file name.  
+`SRC_DIR`: The name of the directory inside of the zip file.
+
+For example to run a Docker container with the exact code of the release 2.0.0:
+
+```
+docker run \
+  -e 
URL="https://www.apache.org/dyn/closer.cgi?filename=beam/2.0.0/apache-beam-2.0.0-source-release.zip&action=download";
 \
+  -e SRC_FILE="apache-beam-2.0.0-source-release.zip" \
+  -e SRC_DIR="apache-beam-2.0.0" \
+  -it beam:openjdk8 /bin/bash
+```
+
+If you want to run a container with a specific version e.g. 2.1.0-RC1:
+
+```
+docker run \
+  -e URL="https://github.com/apache/beam/archive/v2.1.0-RC1.zip"; \
+  -e SRC_FILE="v2.1.0-RC1.zip" \
+  -e SRC_DIR="beam-2.1.0-RC1" \
+  -it beam:openjdk8 /bin/bash
+```
+
+*Notice that SRC_FILE is different from SRC_DIR because github replaces the 'v'
+character in the internal name of the zip*.
+
+To run a container with the source code during a vote:
+
+```
+docker run \
+  -e 
URL="https://dist.apache.org/repos/dist/dev/beam/2.1.0/apache-beam-2.1.0-source-release.zip";
 \
+  -e SRC_FILE="apache-beam-2.1.0-source-release.zip" \
+  -e SRC_DIR="apache-beam-2.1.0" \
+  -it beam:openjdk8 /bin/bash
+```
+
+### Testing in an specific environment with your own source
+
+You can also overwrite the volume containing the source code of Beam from a
+directory in your host machine. This is useful to test your code with different
+versions of Java.
+
+If you have the code in the ~/workspace/beam directory and you want to quickly
+test it with Beam on Java 7 you can do:
+
+```
+docker run -v ~/workspace/beam:/home/user/beam -it beam:openjdk7 /bin/bash
+``

[beam-site] branch mergebot updated (fa0cc80 -> d7e0b34)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from fa0cc80  This closes #318
 add 0ad1d91  Prepare repository for deployment.
 new d3f3543  [BEAM-1534] Add docker images documentation to the website
 new d7e0b34  This closes #297

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/downloads/index.html |  16 +--
 src/_includes/header.html|   1 +
 src/contribute/docker-images.md  | 183 +++
 3 files changed, 192 insertions(+), 8 deletions(-)
 create mode 100644 src/contribute/docker-images.md

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 02/02: This closes #297

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d7e0b34b1897ad766340077604484a4c263c1122
Merge: 0ad1d91 d3f3543
Author: Mergebot 
AuthorDate: Sun Sep 17 23:28:33 2017 +

This closes #297

 src/_includes/header.html   |   1 +
 src/contribute/docker-images.md | 183 
 2 files changed, 184 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/01: Prepare repository for deployment.

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4cd1ee8969ff692b561bfa4dab0db2316a16aaa7
Author: Mergebot 
AuthorDate: Sun Sep 17 23:30:23 2017 +

Prepare repository for deployment.
---
 content/beam/capability/2016/03/17/capability-matrix.html  | 1 +
 content/beam/capability/2016/04/03/presentation-materials.html | 1 +
 content/beam/python/sdk/2016/02/25/python-sdk-now-public.html  | 1 +
 content/beam/release/2016/06/15/first-release.html | 1 +
 content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html   | 1 +
 content/beam/update/website/2016/02/22/beam-has-a-logo.html| 1 +
 content/blog/2016/05/18/splitAtFraction-method.html| 1 +
 content/blog/2016/05/27/where-is-my-pcollection-dot-map.html   | 1 +
 content/blog/2016/06/13/flink-batch-runner-milestone.html  | 1 +
 content/blog/2016/08/03/six-months.html| 1 +
 content/blog/2016/10/20/test-stream.html   | 1 +
 content/blog/2017/01/09/added-apex-runner.html | 1 +
 content/blog/2017/01/10/beam-graduates.html| 1 +
 content/blog/2017/02/01/graduation-media-recap.html| 1 +
 content/blog/2017/02/13/stateful-processing.html   | 1 +
 content/blog/2017/03/16/python-sdk-release.html| 1 +
 content/blog/2017/05/17/beam-first-stable-release.html | 1 +
 content/blog/2017/08/16/splittable-do-fn.html  | 1 +
 content/blog/2017/08/28/timely-processing.html | 1 +
 content/blog/index.html| 1 +
 content/coming-soon.html   | 1 +
 content/contribute/contribution-guide/index.html   | 1 +
 content/contribute/design-principles/index.html| 1 +
 content/contribute/index.html  | 1 +
 content/contribute/logos/index.html| 1 +
 content/contribute/maturity-model/index.html   | 1 +
 content/contribute/presentation-materials/index.html   | 1 +
 content/contribute/ptransform-style-guide/index.html   | 1 +
 content/contribute/release-guide/index.html| 1 +
 content/contribute/runner-guide/index.html | 1 +
 content/contribute/source-repository/index.html| 1 +
 content/contribute/team/index.html | 1 +
 content/contribute/testing/index.html  | 1 +
 content/contribute/work-in-progress/index.html | 1 +
 content/documentation/dsls/sql/index.html  | 1 +
 content/documentation/index.html   | 1 +
 content/documentation/io/authoring-java/index.html | 1 +
 content/documentation/io/authoring-overview/index.html | 1 +
 content/documentation/io/authoring-python/index.html   | 1 +
 content/documentation/io/built-in/hadoop/index.html| 1 +
 content/documentation/io/built-in/index.html   | 1 +
 content/documentation/io/contributing/index.html   | 1 +
 content/documentation/io/io-toc/index.html | 1 +
 content/documentation/io/testing/index.html| 1 +
 content/documentation/pipelines/create-your-pipeline/index.html| 1 +
 content/documentation/pipelines/design-your-pipeline/index.html| 1 +
 content/documentation/pipelines/test-your-pipeline/index.html  | 1 +
 content/documentation/programming-guide/index.html | 1 +
 content/documentation/resources/index.html | 1 +
 content/documentation/runners/apex/index.html  | 1 +
 content/documentation/runners/capability-matrix/index.html | 1 +
 content/documentation/runners/dataflow/index.html  | 1 +
 content/documentation/runners/direct/index.html| 1 +
 content/documentation/runners/flink/index.html | 1 +
 content/documentation/runners/gearpump/index.html  | 1 +
 content/documentation/runners/spark/index.html | 1 +
 content/documentation/sdks/java-extensions/index.html  | 1 +
 content/documentation/sdks/java/index.html | 1 +
 content/documentation/sdks/python-custom-io/index.html | 1 +
 content/documentation/sdks/python-pipeline-dependencies/index.html | 1 +
 content/documentation/sdks/python-type-safety/index.html   | 1 +
 content/documentation/sdks/python/index.html   | 1 +
 content/get-started/beam-overview/index.html   | 1 +

[beam-site] branch asf-site updated (0ad1d91 -> 4cd1ee8)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 0ad1d91  Prepare repository for deployment.
 add d3f3543  [BEAM-1534] Add docker images documentation to the website
 add d7e0b34  This closes #297
 new 4cd1ee8  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../capability/2016/03/17/capability-matrix.html   |   1 +
 .../2016/04/03/presentation-materials.html |   1 +
 .../sdk/2016/02/25/python-sdk-now-public.html  |   1 +
 content/beam/release/2016/06/15/first-release.html |   1 +
 .../2016/10/11/strata-hadoop-world-and-beam.html   |   1 +
 .../update/website/2016/02/22/beam-has-a-logo.html |   1 +
 .../blog/2016/05/18/splitAtFraction-method.html|   1 +
 .../05/27/where-is-my-pcollection-dot-map.html |   1 +
 .../2016/06/13/flink-batch-runner-milestone.html   |   1 +
 content/blog/2016/08/03/six-months.html|   1 +
 content/blog/2016/10/20/test-stream.html   |   1 +
 content/blog/2017/01/09/added-apex-runner.html |   1 +
 content/blog/2017/01/10/beam-graduates.html|   1 +
 .../blog/2017/02/01/graduation-media-recap.html|   1 +
 content/blog/2017/02/13/stateful-processing.html   |   1 +
 content/blog/2017/03/16/python-sdk-release.html|   1 +
 .../blog/2017/05/17/beam-first-stable-release.html |   1 +
 content/blog/2017/08/16/splittable-do-fn.html  |   1 +
 content/blog/2017/08/28/timely-processing.html |   1 +
 content/blog/index.html|   1 +
 content/coming-soon.html   |   1 +
 content/contribute/contribution-guide/index.html   |   1 +
 content/contribute/design-principles/index.html|   1 +
 content/contribute/index.html  |   1 +
 content/contribute/logos/index.html|   1 +
 content/contribute/maturity-model/index.html   |   1 +
 .../contribute/presentation-materials/index.html   |   1 +
 .../contribute/ptransform-style-guide/index.html   |   1 +
 content/contribute/release-guide/index.html|   1 +
 content/contribute/runner-guide/index.html |   1 +
 content/contribute/source-repository/index.html|   1 +
 content/contribute/team/index.html |   1 +
 content/contribute/testing/index.html  |   1 +
 content/contribute/work-in-progress/index.html |   1 +
 content/documentation/dsls/sql/index.html  |   1 +
 content/documentation/index.html   |   1 +
 content/documentation/io/authoring-java/index.html |   1 +
 .../documentation/io/authoring-overview/index.html |   1 +
 .../documentation/io/authoring-python/index.html   |   1 +
 .../documentation/io/built-in/hadoop/index.html|   1 +
 content/documentation/io/built-in/index.html   |   1 +
 content/documentation/io/contributing/index.html   |   1 +
 content/documentation/io/io-toc/index.html |   1 +
 content/documentation/io/testing/index.html|   1 +
 .../pipelines/create-your-pipeline/index.html  |   1 +
 .../pipelines/design-your-pipeline/index.html  |   1 +
 .../pipelines/test-your-pipeline/index.html|   1 +
 content/documentation/programming-guide/index.html |   1 +
 content/documentation/resources/index.html |   1 +
 content/documentation/runners/apex/index.html  |   1 +
 .../runners/capability-matrix/index.html   |   1 +
 content/documentation/runners/dataflow/index.html  |   1 +
 content/documentation/runners/direct/index.html|   1 +
 content/documentation/runners/flink/index.html |   1 +
 content/documentation/runners/gearpump/index.html  |   1 +
 content/documentation/runners/spark/index.html |   1 +
 .../documentation/sdks/java-extensions/index.html  |   1 +
 content/documentation/sdks/java/index.html |   1 +
 .../documentation/sdks/python-custom-io/index.html |   1 +
 .../sdks/python-pipeline-dependencies/index.html   |   1 +
 .../sdks/python-type-safety/index.html |   1 +
 content/documentation/sdks/python/index.html   |   1 +
 content/get-started/beam-overview/index.html   |   1 +
 content/get-started/downloads/index.html   |   1 +
 content/get-started/index.html |   1 +
 .../get-started/mobile-gaming-example/index.html   |   1 +
 content/get-started/quickstart-java/index.html |   1 +
 content/get-started/quickstart-py/index.html   |   1 +
 content/get-started/support/index.html |   1 +
 content/get-started/wordcount-example/index.html   |   1 +
 content/index.html |   1 +
 content/privacy_policy/index.html  |   1 +
 src/_includes/header.h

[beam-site] 01/02: [BEAM-2726] Fix typo in programming guide

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ea580a43cdf9814f7e32c1a200124bcfabd49b76
Author: Lukasz Cwik 
AuthorDate: Fri Aug 4 06:52:30 2017 -0700

[BEAM-2726] Fix typo in programming guide
---
 src/documentation/programming-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 70c3785..061a36a 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -670,7 +670,7 @@ pc = ...
 
 # Global windowing:
 
-If your input `PCollection` uses the default global windowing, the default 
behavior is to return a `PCollection` containing one item. That item's value 
comes from the accumulator in the combine function that you specified when 
applying `Combine`. For example, the Beam provided sum combine function returns 
a zero value (the sum of an empty input), while the min combine function 
returns a maximal or infinite value.
+If your input `PCollection` uses the default global windowing, the default 
behavior is to return a `PCollection` containing one item. That item's value 
comes from the accumulator in the combine function that you specified when 
applying `Combine`. For example, the Beam provided sum combine function returns 
a zero value (the sum of an empty input), while the max combine function 
returns a maximal or infinite value.
 
 To have `Combine` instead return an empty `PCollection` if the input is empty, 
specify `.withoutDefaults` when you apply your `Combine` transform, as in the 
following code example:
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/02: This closes #280

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 67fa76b9a13bd654e035c21e3a0e96f3f98127f3
Merge: 4cd1ee8 ea580a4
Author: Mergebot 
AuthorDate: Sun Sep 17 23:30:53 2017 +

This closes #280

 src/documentation/programming-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (d7e0b34 -> 67fa76b)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from d7e0b34  This closes #297
 add 4cd1ee8  Prepare repository for deployment.
 new ea580a4  [BEAM-2726] Fix typo in programming guide
 new 67fa76b  This closes #280

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/beam/capability/2016/03/17/capability-matrix.html  | 1 +
 content/beam/capability/2016/04/03/presentation-materials.html | 1 +
 content/beam/python/sdk/2016/02/25/python-sdk-now-public.html  | 1 +
 content/beam/release/2016/06/15/first-release.html | 1 +
 content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html   | 1 +
 content/beam/update/website/2016/02/22/beam-has-a-logo.html| 1 +
 content/blog/2016/05/18/splitAtFraction-method.html| 1 +
 content/blog/2016/05/27/where-is-my-pcollection-dot-map.html   | 1 +
 content/blog/2016/06/13/flink-batch-runner-milestone.html  | 1 +
 content/blog/2016/08/03/six-months.html| 1 +
 content/blog/2016/10/20/test-stream.html   | 1 +
 content/blog/2017/01/09/added-apex-runner.html | 1 +
 content/blog/2017/01/10/beam-graduates.html| 1 +
 content/blog/2017/02/01/graduation-media-recap.html| 1 +
 content/blog/2017/02/13/stateful-processing.html   | 1 +
 content/blog/2017/03/16/python-sdk-release.html| 1 +
 content/blog/2017/05/17/beam-first-stable-release.html | 1 +
 content/blog/2017/08/16/splittable-do-fn.html  | 1 +
 content/blog/2017/08/28/timely-processing.html | 1 +
 content/blog/index.html| 1 +
 content/coming-soon.html   | 1 +
 content/contribute/contribution-guide/index.html   | 1 +
 content/contribute/design-principles/index.html| 1 +
 content/contribute/index.html  | 1 +
 content/contribute/logos/index.html| 1 +
 content/contribute/maturity-model/index.html   | 1 +
 content/contribute/presentation-materials/index.html   | 1 +
 content/contribute/ptransform-style-guide/index.html   | 1 +
 content/contribute/release-guide/index.html| 1 +
 content/contribute/runner-guide/index.html | 1 +
 content/contribute/source-repository/index.html| 1 +
 content/contribute/team/index.html | 1 +
 content/contribute/testing/index.html  | 1 +
 content/contribute/work-in-progress/index.html | 1 +
 content/documentation/dsls/sql/index.html  | 1 +
 content/documentation/index.html   | 1 +
 content/documentation/io/authoring-java/index.html | 1 +
 content/documentation/io/authoring-overview/index.html | 1 +
 content/documentation/io/authoring-python/index.html   | 1 +
 content/documentation/io/built-in/hadoop/index.html| 1 +
 content/documentation/io/built-in/index.html   | 1 +
 content/documentation/io/contributing/index.html   | 1 +
 content/documentation/io/io-toc/index.html | 1 +
 content/documentation/io/testing/index.html| 1 +
 content/documentation/pipelines/create-your-pipeline/index.html| 1 +
 content/documentation/pipelines/design-your-pipeline/index.html| 1 +
 content/documentation/pipelines/test-your-pipeline/index.html  | 1 +
 content/documentation/programming-guide/index.html | 1 +
 content/documentation/resources/index.html | 1 +
 content/documentation/runners/apex/index.html  | 1 +
 content/documentation/runners/capability-matrix/index.html | 1 +
 content/documentation/runners/dataflow/index.html  | 1 +
 content/documentation/runners/direct/index.html| 1 +
 content/documentation/runners/flink/index.html | 1 +
 content/documentation/runners/gearpump/index.html  | 1 +
 content/documentation/runners/spark/index.html | 1 +
 content/documentation/sdks/java-extensions/index.html  | 1 +
 content/documentation/sdks/java/index.html | 1 +
 content/documentation/sdks/python-custom-io/index.html | 1 +
 content/doc

[beam-site] branch asf-site updated (4cd1ee8 -> 1fe5f55)

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4cd1ee8  Prepare repository for deployment.
 add ea580a4  [BEAM-2726] Fix typo in programming guide
 add 67fa76b  This closes #280
 new 1fe5f55  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/programming-guide/index.html | 2 +-
 src/documentation/programming-guide.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

2017-09-17 Thread mergebot-role

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 1fe5f5593f4e0e10b6bd7b3ab460846f74d8c041
Author: Mergebot 
AuthorDate: Sun Sep 17 23:32:54 2017 +

Prepare repository for deployment.
---
 content/documentation/programming-guide/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index 1d5bd92..0234c78 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -892,7 +892,7 @@ guest, [[], [order4]]
 
 Global windowing:
 
-If your input PCollection uses the 
default global windowing, the default behavior is to return a PCollection containing one item. That item’s 
value comes from the accumulator in the combine function that you specified 
when applying Combine. For example, the 
Beam provided sum combine function returns a zero value (the sum of an empty 
input), while the min combine functio [...]
+If your input PCollection uses the 
default global windowing, the default behavior is to return a PCollection containing one item. That item’s 
value comes from the accumulator in the combine function that you specified 
when applying Combine. For example, the 
Beam provided sum combine function returns a zero value (the sum of an empty 
input), while the max combine functio [...]
 
 To have Combine instead return an 
empty PCollection if the input is empty, 
specify .withoutDefaults when you apply 
your Combine transform, as in the 
following code example:
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4812

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 3.63 MB...]
  File 
"
 line 25, in 
from apitools.base.protorpclite import messages
  File 
"
 line 1165, in 
class Field(six.with_metaclass(_FieldMeta, object)):
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) 
subclass of the metaclasses of all its bases
ERROR: InvocationError: 
'
 apache_beam/examples/complete/autocomplete_test.py'
___ summary 
ERROR:   docs: commands failed
  lint: commands succeeded
  py27: commands succeeded
  py27cython: commands succeeded
ERROR:   py27gcp: commands failed
2017-09-18T01:14:03.382 [ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit 
value: 1)
at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at 
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at jenkins.maven3.agent.Maven33Main.launch(Maven33Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:139)
at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[JENKINS] Archiving disabled
2017-09-18T01:14:04.292 [INFO]  
   
2017-09-18T01:14:04.292 [INFO] 

2017-09-18T01:14:04.292 [INFO] Skipping Apache Beam :: Parent
2017-09-18T01:14:04.292 [INFO] This project has been banned from the bui

Build failed in Jenkins: beam_PostCommit_Python_Verify #3153

2017-09-17 Thread Apache Jenkins Server

See 


--
[...truncated 43.99 KB...]
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.2.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.2.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.2.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.2.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-8kHjya-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpG61Ngypip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-8kHjya-build/setup.py", line 198, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-8kHjya-build/setup.py", line 138, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, 
mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: protobuf 3.4

40 matches

Mail list logo