[jira] [Work logged] (BEAM-3485) CassandraIO.read() splitting produces invalid queries

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3485?focusedWorklogId=95375&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95375
 ]

ASF GitHub Bot logged work on BEAM-3485:


Author: ASF GitHub Bot
Created on: 26/Apr/18 06:57
Start Date: 26/Apr/18 06:57
Worklog Time Spent: 10m 
  Work Description: adejanovski commented on a change in pull request 
#5124: [BEAM-3485] Fix split generation for Cassandra clusters
URL: https://github.com/apache/beam/pull/5124#discussion_r184286740
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/SplitGenerator.java
 ##
 @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.google.common.collect.Lists;
+import java.math.BigInteger;
+import java.util.List;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Splits given Cassandra table's token range into splits.
+ */
+final class SplitGenerator {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(SplitGenerator.class);
+
+  private final String partitioner;
+  private final BigInteger rangeMin;
+  private final BigInteger rangeMax;
+  private final BigInteger rangeSize;
+
+  SplitGenerator(String partitioner){
+rangeMin = getRangeMin(partitioner);
+rangeMax = getRangeMax(partitioner);
+rangeSize = getRangeSize(partitioner);
+this.partitioner = partitioner;
+  }
+
+  SplitGenerator(BigInteger rangeMin, BigInteger rangeMax) {
+this.rangeMin = rangeMin;
+this.rangeMax = rangeMax;
+rangeSize = rangeMax.subtract(rangeMin).add(BigInteger.ONE);
+partitioner = "(" + rangeMin + "," + rangeMax + ")";
+  }
+
+  static BigInteger getRangeMin(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return BigInteger.ZERO;
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).negate();
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeMax(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return new BigInteger("2").pow(127).subtract(BigInteger.ONE);
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).subtract(BigInteger.ONE);
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeSize(String partitioner) {
+return 
getRangeMax(partitioner).subtract(getRangeMin(partitioner)).add(BigInteger.ONE);
+  }
+
+  static BigInteger max(BigInteger big0, BigInteger big1) {
+return greaterThan(big0, big1) ? big0 : big1;
+  }
+
+  static BigInteger min(BigInteger big0, BigInteger big1) {
+return lowerThan(big0, big1) ? big0 : big1;
+  }
+
+  static boolean lowerThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) < 0;
+  }
+
+  static boolean lowerThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) <= 0;
+  }
+
+  static boolean greaterThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) > 0;
+  }
+
+  static boolean greaterThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) >= 0;
+  }
+
+  /**
+   * Given big0 properly ordered list of tokens, compute at least {@code 
totalSplitCount} splits.
+   *
+   * @param totalSplitCount requested total amount of splits. This function 
may generate
+   *more splits.
+   * @param ringTokens list of all start tokens in big0 cluster. They have to 
be in ring order.
+   * @return big0 list containing at least {@code totalSplitCount} splits.
+   */
+  List generateSplits(long totalSplitCount, List 
ringTokens) {
+
+int tokenRangeCount = ringTokens.size();
 
 Review comment:
   @iemejia : I've just pushed the fix.
   I hadn't handled the wrapping range correctly (the one that covers both the 
end 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #195

[Apache Jenkins Server]

See 


--
[...truncated 18.89 MB...]
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.293Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/GroupByKey/Write into 
PAssert$3/CreateActual/GatherPanes/GroupByKey/Reify
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.318Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Window.Into()/Window.Assign into 
PAssert$3/CreateActual/GatherPanes/WithKeys/AddKeys/Map
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.346Z: Unzipping flatten s18-u63 for input 
s19.output-c61
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.375Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.400Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.427Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.457Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.480Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.508Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.531Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.554Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.582Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Apr 26, 2018 6:47:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T06:47:20.605Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Si

Jenkins build is back to normal : beam_PerformanceTests_JDBC #498

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #421

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #96

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[swegner] Upgrade Gradle to 4.7

[swegner] Upgrade Gradle protobuf plugin to latest.

[tgroh] fixup! Add OutputDeduplicator

--
[...truncated 865.80 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.49.14:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.49.14:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.49.14:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.M

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #423

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #94

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[swegner] Upgrade Gradle to 4.7

[swegner] Upgrade Gradle protobuf plugin to latest.

[tgroh] fixup! Add OutputDeduplicator

--
[...truncated 944.74 KB...]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy63.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy62.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy63.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFil

Build failed in Jenkins: beam_PerformanceTests_Python #1194

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[swegner] Upgrade Gradle to 4.7

[swegner] Upgrade Gradle protobuf plugin to latest.

[tgroh] fixup! Add OutputDeduplicator

--
[...truncated 816.79 KB...]
[INFO] Apache Beam :: Model ... SUCCESS [  0.035 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  8.533 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.907 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  3.906 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.132 s]
[INFO] Apache Beam :: SDKs :: Go .. SUCCESS [ 31.284 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SUCCESS [ 22.075 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  0.154 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [ 26.405 s]
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SUCCESS [  3.083 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [  0.050 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SUCCESS [  3.428 s]
[INFO] Apache Beam :: Runners . SUCCESS [  0.060 s]
[INFO] Apache Beam :: Runners :: Core Construction Java ... SUCCESS [  4.812 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [  4.796 s]
[INFO] Apache Beam :: SDKs :: Java :: Harness . SUCCESS [  8.793 s]
[INFO] Apache Beam :: SDKs :: Java :: Container ... SUCCESS [  8.932 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  0.054 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SUCCESS [  
2.953 s]
[INFO] Apache Beam :: Runners :: Local Java Core .. SUCCESS [  1.606 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [ 11.405 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SUCCESS [  2.618 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Common  SUCCESS [  1.865 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Cassandra . SUCCESS [  2.769 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch . SUCCESS [  1.959 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests SUCCESS [  
0.553 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: Common 
SUCCESS [  0.638 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 2.x SUCCESS 
[  1.601 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 5.x SUCCESS 
[  1.990 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: XML ... SUCCESS [  4.223 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Protobuf SUCCESS [  2.117 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS [  
4.934 s]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [ 11.940 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: File-based-io-tests SUCCESS [  
2.123 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Common . SUCCESS [  3.187 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop File System SUCCESS [  3.302 
s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [  4.767 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Input Format SUCCESS [  
8.049 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HBase . SUCCESS [  6.612 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HCatalog .. SUCCESS [  7.833 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [  3.272 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [  2.675 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [  1.614 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [  3.357 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MQTT .. SUCCESS [  2.192 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Redis . SUCCESS [  2.052 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Solr .. SUCCESS [  4.998 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Tika .. SUCCESS [  3.233 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [  0.028 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter SUCCESS [  
6.954 s]
[INFO] Apache Beam :: Examples  SUCCESS [  0.028 s]
[INFO] Apache Beam :: Examples :: Java  SUCCESS [  3.841 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples SUCCESS [ 
28.156 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Jackson SUCCESS [  1.933 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library SUCCESS [  
2.213 s]
[INFO] Apache Beam :: SDKs :: Java

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #95

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[swegner] Upgrade Gradle to 4.7

[swegner] Upgrade Gradle protobuf plugin to latest.

[tgroh] fixup! Add OutputDeduplicator

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ac7bfb86ed241b46dbf5d249ea4464478f3fb180 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ac7bfb86ed241b46dbf5d249ea4464478f3fb180
Commit message: "Merge pull request #4977: Deduplicate Unzipped Flattens after 
Pipeline Fusion"
 > git rev-list --no-walk bbabb4202cdbe234ab7be9a1cda57f0e8e226357 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1825323276220131695.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7234801034451994732.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7817746783588052647.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1524718941628
namespace "filebasedioithdfs-1524718941628" created
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins978264145741401057.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1524718941628
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8934393002983009651.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8417668267297113899.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8696566115428540475.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /home/jenkins/jenkin...DFS/.env/bin/python2 - 
setuptools pkg_resources pip wheel:
  Collecting setuptools
  Using cached 
https://files.pythonhosted.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl
Collecting pkg_resources
Collecting pip
  Using cached 
https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Collecting wheel
Exception:
Traceback (most recent call last):
  File 
"/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", 
line 209, in main
status = self.run(options, args)
  File 
"/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py",
 line 328, in run
wb.build(autobuilding=True)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/wheel.py", 
line 748, in build

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #187

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[swegner] Upgrade Gradle to 4.7

[swegner] Upgrade Gradle protobuf plugin to latest.

[tgroh] fixup! Add OutputDeduplicator

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam24 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ac7bfb86ed241b46dbf5d249ea4464478f3fb180 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ac7bfb86ed241b46dbf5d249ea4464478f3fb180
Commit message: "Merge pull request #4977: Deduplicate Unzipped Flattens after 
Pipeline Fusion"
 > git rev-list --no-walk bbabb4202cdbe234ab7be9a1cda57f0e8e226357 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins4763056288901553429.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins646493975096423166.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins8546555418143381614.sh
+ kubectl 
--kubeconfig=
 create namespace hadoopinputformatioit-1524718955984
namespace "hadoopinputformatioit-1524718955984" created
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins2905206499834172386.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=hadoopinputformatioit-1524718955984
error: open /home/jenkins/.kube/config.lock: file exists
Build step 'Execute shell' marked build as failure

[jira] [Work logged] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3587?focusedWorklogId=95358&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95358
 ]

ASF GitHub Bot logged work on BEAM-3587:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:39
Start Date: 26/Apr/18 05:39
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #384: [BEAM-3587] Add a note 
to Gradle shadowJar for merge service files
URL: https://github.com/apache/beam-site/pull/384#issuecomment-384519643
 
 
   Is this still relevant with all of the Gradle changes recently?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95358)
Time Spent: 0.5h  (was: 20m)

> User reports TextIO failure in FlinkRunner on master
> 
>
> Key: BEAM-3587
> URL: https://issues.apache.org/jira/browse/BEAM-3587
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Attachments: screen1.png, screen2.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Reported here: 
> [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink 
> cluster, using the latest Beam git revision (ff37337). The job fails to start 
> with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not 
> supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  
> [https://github.com/pelletier/beam-flink-example]";
> My initial thoughts: I have a guess that this has to do with switching to 
> running from a portable pipeline representation, and it looks like there's a 
> non-composite transform with an empty URN and it threw a bad error message. 
> We can try to root cause but may also mitigate short-term by removing the 
> round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - 
> they only run on a local Flink, yet this seems to be a translation issue that 
> would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the 
> bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4119) Update I/O testing documentation for Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4119?focusedWorklogId=95347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95347
 ]

ASF GitHub Bot logged work on BEAM-4119:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:09
Start Date: 26/Apr/18 05:09
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #425: [BEAM-4119] Update i/o 
testing documentation to use gradle
URL: https://github.com/apache/beam-site/pull/425#issuecomment-384514650
 
 
   Hi there @lgajowy , thanks for the PR! the content/ directory on the website 
is autogenerated from the files in the src/ directory after a merge, so your 
changes here would be overwritten. could you please move your changes into the 
associated file in the src/ directory (src/documentation/io/testing.md) and 
revert the content/ file? thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95347)
Time Spent: 50m  (was: 40m)

> Update I/O testing documentation for Gradle
> ---
>
> Key: BEAM-4119
> URL: https://issues.apache.org/jira/browse/BEAM-4119
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Łukasz Gajowy
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The performance testing framework is now integrated into the Gradle build 
> with [PR/5003|https://github.com/apache/beam/pull/5003]. The website needs to 
> be updated to reflect the new commands: 
> [https://beam.apache.org/documentation/io/testing/] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95346
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:04
Start Date: 26/Apr/18 05:04
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513986
 
 
   the robot likes me right now @kennknowles  ;)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95346)
Time Spent: 10h 40m  (was: 10.5h)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95345
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:03
Start Date: 26/Apr/18 05:03
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/documentation/sdks/nexmark.md 
b/src/documentation/sdks/nexmark.md
index 5e86cd782..d73943a4f 100644
--- a/src/documentation/sdks/nexmark.md
+++ b/src/documentation/sdks/nexmark.md
@@ -9,12 +9,14 @@ permalink: /documentation/sdks/java/nexmark/
 ## What it is
 
 Nexmark is a suite of pipelines inspired by the 'continuous data stream'
-queries in [Nexmark research 
paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
+queries in [Nexmark research
+paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
 
-These are multiple queries over a three entities model representing on online 
auction system:
+These are multiple queries over a three entities model representing on online
+auction system:
 
- - **Person** represents a person submitting an item for auction and/or making 
a bid
-on an auction.
+ - **Person** represents a person submitting an item for auction and/or making
+   a bid on an auction.
  - **Auction** represents an item under auction.
  - **Bid** represents a bid for an item under auction.
 
@@ -62,11 +64,14 @@ We have augmented the original queries with five more:
   queries.
 
 ## Benchmark workload configuration
-Here are some of the knobs of the benchmark workload (see 
[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
+
+Here are some of the knobs of the benchmark workload (see
+[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
 
 These configuration items can be passed to the launch command line.
 
 ### Events generation (defaults)
+
 * 100 000 events generated
 * 100 generator threads
 * Event rate in SIN curve
@@ -76,21 +81,26 @@ These configuration items can be passed to the launch 
command line.
 * 1000 concurrent persons bidding / creating auctions
 
 ### Windows (defaults)
+
 * size 10s
 * sliding period 5s
 * watermark hold for 0s
 
 ### Events Proportions (defaults)
+
 * Hot Auctions = ½
 * Hot Bidders =¼
 * Hot Sellers=¼
 
 ### Technical
+
 * Artificial CPU load
 * Artificial IO load
 
 ## Nexmark output
-Here is an example output of the Nexmark benchmark run in streaming mode with 
the SMOKE suite on the (local) direct runner:
+
+Here is an example output of the Nexmark benchmark run in streaming mode with
+the SMOKE suite on the (local) direct runner:
 
 
 Performance:
@@ -112,17 +122,19 @@ Performance:
 
 ## Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available profiles 
are:
-
-direct-runner
-spark-runner
-flink-runner
-apex-runner
+The Nexmark launcher accepts the `--runner` argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-The runner must also be specified like in any other Beam pipeline using:
+When running via Gradle, the following two parameters control the execution:
 
---runner
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
+-P nexmark.runner
+   The Gradle project name of the runner, such as 
":beam-runners-direct-java" or
+   ":beam-runners-flink. The project names can be found in the root
+`settings.gradle`.
 
 Test data is deterministically synthesized on demand. The test
 data may be synthesized in the same pipeline as the query itself,
@@ -417,124 +429,192 @@ Yet to come
 
 ### Running SMOKE suite on the DirectRunner (local)
 
+The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`.
+Here we do it for maximum clarity.
+
+The direct runner does not have separate batch and streaming modes, but the
+Nexmark launch does.
+
+These parameters leave on many of the DirectRunner's extra safety checks so the
+SMOKE suite can make sure there is nothing broken in the Nexmark suite.
+
 Batch Mode:
 
-mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
-Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE 
--streaming=fal

[beam-site] branch asf-site updated (9193579 -> 8c27c4b)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 9193579  Prepare repository for deployment.
 add d41a892  Update Nexmark launch instructions for Gradle
 add 75cae4f  This closes #418
 new 8c27c4b  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/sdks/java/nexmark/index.html | 259 +---
 src/documentation/sdks/nexmark.md  | 270 +
 2 files changed, 348 insertions(+), 181 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8c27c4bdeb842dc72a61e006ceb1011efe23098c
Author: Mergebot 
AuthorDate: Wed Apr 25 22:03:42 2018 -0700

Prepare repository for deployment.
---
 content/documentation/sdks/java/nexmark/index.html | 259 ++---
 1 file changed, 173 insertions(+), 86 deletions(-)

diff --git a/content/documentation/sdks/java/nexmark/index.html 
b/content/documentation/sdks/java/nexmark/index.html
index acbaaa3..e490497 100644
--- a/content/documentation/sdks/java/nexmark/index.html
+++ b/content/documentation/sdks/java/nexmark/index.html
@@ -180,13 +180,15 @@
 What it is
 
 Nexmark is a suite of pipelines inspired by the ‘continuous data stream’
-queries in http://datalab.cs.pdx.edu/niagaraST/NEXMark/";>Nexmark 
research paper
+queries in http://datalab.cs.pdx.edu/niagaraST/NEXMark/";>Nexmark 
research
+paper
 
-These are multiple queries over a three entities model representing on 
online auction system:
+These are multiple queries over a three entities model representing on 
online
+auction system:
 
 
-  Person represents a person submitting an item for 
auction and/or making a bid
- on an auction.
+  Person represents a person submitting an item for 
auction and/or making
+a bid on an auction.
   Auction represents an item under auction.
   Bid represents a bid for an item under auction.
 
@@ -239,11 +241,14 @@ queries.
 
 
 Benchmark workload configuration
-Here are some of the knobs of the benchmark workload (see https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java";>NexmarkConfiguration.java).
+
+Here are some of the knobs of the benchmark workload (see
+https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java";>NexmarkConfiguration.java).
 
 These configuration items can be passed to the launch command line.
 
 Events generation (defaults)
+
 
   100 000 events generated
   100 generator threads
@@ -255,6 +260,7 @@ queries.
 
 
 Windows (defaults)
+
 
   size 10s
   sliding period 5s
@@ -262,6 +268,7 @@ queries.
 
 
 Events Proportions (defaults)
+
 
   Hot Auctions = ½
   Hot Bidders =¼
@@ -269,13 +276,16 @@ queries.
 
 
 Technical
+
 
   Artificial CPU load
   Artificial IO load
 
 
 Nexmark output
-Here is an example output of the Nexmark benchmark run in streaming mode 
with the SMOKE suite on the (local) direct runner:
+
+Here is an example output of the Nexmark benchmark run in streaming mode 
with
+the SMOKE suite on the (local) direct runner:
 
 
 Performance:
@@ -297,18 +307,19 @@ Performance:
 
 Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available 
profiles are:
+The Nexmark launcher accepts the --runner argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-direct-runner
-spark-runner
-flink-runner
-apex-runner
-
-
+When running via Gradle, the following two parameters control the 
execution:
 
-The runner must also be specified like in any other Beam pipeline using:
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
---runner
+-P nexmark.runner
+The Gradle project name of the runner, such as ":beam-runners-direct-java" or
+":beam-runners-flink. The project names can be found in the root
+`settings.gradle`.
 
 
 
@@ -625,29 +636,76 @@ or may be published to Pub/Sub.
 
 Running SMOKE suite on 
the DirectRunner (local)
 
+The DirectRunner is default, so it is not required to pass -Pnexmark.runner.
+Here we do it for maximum clarity.
+
+The direct runner does not have separate batch and streaming modes, but the
+Nexmark launch does.
+
+These parameters leave on many of the DirectRunner’s extra safety checks so 
the
+SMOKE suite can make sure there is nothing broken in the Nexmark suite.
+
 Batch Mode:
 
-mvn exec:java 
-Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
-Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=false 
--manageResources=false --monitorJobs=true --enforceEncodability=true 
--enforceImmutability=true"
+./gradlew 
:beam-sdks-java-nexmark:run \
+-Pnexmark.runner=":beam-runners-direct-java" \
+-Pnexmark.args="
+--runner=DirectRunner
+--streaming=false
+--suite=SMOKE
+--manageResources=false
+--monitorJobs=true
+--enforceEncodability=true
+--enforceImmutability=true"
 
 
 
 Streaming Mode:
 
-mvn exec:java 
-Dexec.mainClass=org.apache.beam.sdk.nexmark.Main -Pdirect-runner 
-Dexec.args="--runner=DirectRunner --suite=SMOKE --streaming=true 
--manageResources=false --monitorJobs=true --enforceEncodability=true 
--enforceImmutability=true"
+./gradlew 
:beam-sdks-j

[jira] [Work logged] (BEAM-4119) Update I/O testing documentation for Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4119?focusedWorklogId=95344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95344
 ]

ASF GitHub Bot logged work on BEAM-4119:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:01
Start Date: 26/Apr/18 05:01
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #425: [BEAM-4119] Update i/o 
testing documentation to use gradle
URL: https://github.com/apache/beam-site/pull/425#issuecomment-384513555
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95344)
Time Spent: 40m  (was: 0.5h)

> Update I/O testing documentation for Gradle
> ---
>
> Key: BEAM-4119
> URL: https://issues.apache.org/jira/browse/BEAM-4119
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Łukasz Gajowy
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The performance testing framework is now integrated into the Gradle build 
> with [PR/5003|https://github.com/apache/beam/pull/5003]. The website needs to 
> be updated to reflect the new commands: 
> [https://beam.apache.org/documentation/io/testing/] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (7cc5dfd -> 75cae4f)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 7cc5dfd  This closes #418
 discard 523bbd0  Update Nexmark launch instructions for Gradle
 new d41a892  Update Nexmark launch instructions for Gradle
 new 75cae4f  This closes #418

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7cc5dfd)
\
 N -- N -- N   refs/heads/mergebot (75cae4f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #418

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 75cae4fb3d55de2bb508ed53db27e75bbd858e08
Merge: 9193579 d41a892
Author: Mergebot 
AuthorDate: Wed Apr 25 22:01:15 2018 -0700

This closes #418

 src/documentation/sdks/nexmark.md | 270 --
 1 file changed, 175 insertions(+), 95 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Update Nexmark launch instructions for Gradle

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d41a8927c4d2ca640f9fdd2abc5fe5ab90e94bf5
Author: Kenneth Knowles 
AuthorDate: Wed Apr 11 11:58:59 2018 -0700

Update Nexmark launch instructions for Gradle
---
 src/documentation/sdks/nexmark.md | 270 --
 1 file changed, 175 insertions(+), 95 deletions(-)

diff --git a/src/documentation/sdks/nexmark.md 
b/src/documentation/sdks/nexmark.md
index 5e86cd7..d73943a 100644
--- a/src/documentation/sdks/nexmark.md
+++ b/src/documentation/sdks/nexmark.md
@@ -9,12 +9,14 @@ permalink: /documentation/sdks/java/nexmark/
 ## What it is
 
 Nexmark is a suite of pipelines inspired by the 'continuous data stream'
-queries in [Nexmark research 
paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
+queries in [Nexmark research
+paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
 
-These are multiple queries over a three entities model representing on online 
auction system:
+These are multiple queries over a three entities model representing on online
+auction system:
 
- - **Person** represents a person submitting an item for auction and/or making 
a bid
-on an auction.
+ - **Person** represents a person submitting an item for auction and/or making
+   a bid on an auction.
  - **Auction** represents an item under auction.
  - **Bid** represents a bid for an item under auction.
 
@@ -62,11 +64,14 @@ We have augmented the original queries with five more:
   queries.
 
 ## Benchmark workload configuration
-Here are some of the knobs of the benchmark workload (see 
[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
+
+Here are some of the knobs of the benchmark workload (see
+[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
 
 These configuration items can be passed to the launch command line.
 
 ### Events generation (defaults)
+
 * 100 000 events generated
 * 100 generator threads
 * Event rate in SIN curve
@@ -76,21 +81,26 @@ These configuration items can be passed to the launch 
command line.
 * 1000 concurrent persons bidding / creating auctions
 
 ### Windows (defaults)
+
 * size 10s
 * sliding period 5s
 * watermark hold for 0s
 
 ### Events Proportions (defaults)
+
 * Hot Auctions = ½
 * Hot Bidders =¼
 * Hot Sellers=¼
 
 ### Technical
+
 * Artificial CPU load
 * Artificial IO load
 
 ## Nexmark output
-Here is an example output of the Nexmark benchmark run in streaming mode with 
the SMOKE suite on the (local) direct runner:
+
+Here is an example output of the Nexmark benchmark run in streaming mode with
+the SMOKE suite on the (local) direct runner:
 
 
 Performance:
@@ -112,17 +122,19 @@ Performance:
 
 ## Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available profiles 
are:
-
-direct-runner
-spark-runner
-flink-runner
-apex-runner
+The Nexmark launcher accepts the `--runner` argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-The runner must also be specified like in any other Beam pipeline using:
+When running via Gradle, the following two parameters control the execution:
 
---runner
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
+-P nexmark.runner
+   The Gradle project name of the runner, such as 
":beam-runners-direct-java" or
+   ":beam-runners-flink. The project names can be found in the root
+`settings.gradle`.
 
 Test data is deterministically synthesized on demand. The test
 data may be synthesized in the same pipeline as the query itself,
@@ -417,124 +429,192 @@ Yet to come
 
 ### Running SMOKE suite on the DirectRunner (local)
 
+The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`.
+Here we do it for maximum clarity.
+
+The direct runner does not have separate batch and streaming modes, but the
+Nexmark launch does.
+
+These parameters leave on many of the DirectRunner's extra safety checks so the
+SMOKE suite can make sure there is nothing broken in the Nexmark suite.
+
 Batch Mode:
 
-mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
-Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE 
--streaming=false --manageResources=false --monitorJobs=true 
--enforceEncodability=true --enforceImmutability=true"
+./gradlew :beam-sdks-java-nexmark:run \
+-Pnexmark.runner=":beam-runners-direct-java" \
+-Pnexmark.args="
+--runner=DirectRunner
+--streaming=false
+--suite=SMOKE
+--manageResources=false

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95343&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95343
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 05:00
Start Date: 26/Apr/18 05:00
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513438
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95343)
Time Spent: 10h 20m  (was: 10h 10m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95342&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95342
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 04:59
Start Date: 26/Apr/18 04:59
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#issuecomment-384513328
 
 
   Error: PR failed in verification; check the Jenkins job for more information.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95342)
Time Spent: 10h 10m  (was: 10h)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 02/02: This closes #418

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7cc5dfd3be9925553b71b76f39c1bd66533c000a
Merge: 9193579 523bbd0
Author: Mergebot 
AuthorDate: Wed Apr 25 21:56:49 2018 -0700

This closes #418

 src/documentation/sdks/nexmark.md | 270 --
 1 file changed, 175 insertions(+), 95 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Update Nexmark launch instructions for Gradle

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 523bbd005615b44dfbb86384b07ec6c5e7818075
Author: Kenneth Knowles 
AuthorDate: Wed Apr 11 11:58:59 2018 -0700

Update Nexmark launch instructions for Gradle
---
 src/documentation/sdks/nexmark.md | 270 --
 1 file changed, 175 insertions(+), 95 deletions(-)

diff --git a/src/documentation/sdks/nexmark.md 
b/src/documentation/sdks/nexmark.md
index 5e86cd7..d73943a 100644
--- a/src/documentation/sdks/nexmark.md
+++ b/src/documentation/sdks/nexmark.md
@@ -9,12 +9,14 @@ permalink: /documentation/sdks/java/nexmark/
 ## What it is
 
 Nexmark is a suite of pipelines inspired by the 'continuous data stream'
-queries in [Nexmark research 
paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
+queries in [Nexmark research
+paper](http://datalab.cs.pdx.edu/niagaraST/NEXMark/)
 
-These are multiple queries over a three entities model representing on online 
auction system:
+These are multiple queries over a three entities model representing on online
+auction system:
 
- - **Person** represents a person submitting an item for auction and/or making 
a bid
-on an auction.
+ - **Person** represents a person submitting an item for auction and/or making
+   a bid on an auction.
  - **Auction** represents an item under auction.
  - **Bid** represents a bid for an item under auction.
 
@@ -62,11 +64,14 @@ We have augmented the original queries with five more:
   queries.
 
 ## Benchmark workload configuration
-Here are some of the knobs of the benchmark workload (see 
[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
+
+Here are some of the knobs of the benchmark workload (see
+[NexmarkConfiguration.java](https://github.com/apache/beam/blob/master/sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java)).
 
 These configuration items can be passed to the launch command line.
 
 ### Events generation (defaults)
+
 * 100 000 events generated
 * 100 generator threads
 * Event rate in SIN curve
@@ -76,21 +81,26 @@ These configuration items can be passed to the launch 
command line.
 * 1000 concurrent persons bidding / creating auctions
 
 ### Windows (defaults)
+
 * size 10s
 * sliding period 5s
 * watermark hold for 0s
 
 ### Events Proportions (defaults)
+
 * Hot Auctions = ½
 * Hot Bidders =¼
 * Hot Sellers=¼
 
 ### Technical
+
 * Artificial CPU load
 * Artificial IO load
 
 ## Nexmark output
-Here is an example output of the Nexmark benchmark run in streaming mode with 
the SMOKE suite on the (local) direct runner:
+
+Here is an example output of the Nexmark benchmark run in streaming mode with
+the SMOKE suite on the (local) direct runner:
 
 
 Performance:
@@ -112,17 +122,19 @@ Performance:
 
 ## Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available profiles 
are:
-
-direct-runner
-spark-runner
-flink-runner
-apex-runner
+The Nexmark launcher accepts the `--runner` argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-The runner must also be specified like in any other Beam pipeline using:
+When running via Gradle, the following two parameters control the execution:
 
---runner
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
+-P nexmark.runner
+   The Gradle project name of the runner, such as 
":beam-runners-direct-java" or
+   ":beam-runners-flink. The project names can be found in the root
+`settings.gradle`.
 
 Test data is deterministically synthesized on demand. The test
 data may be synthesized in the same pipeline as the query itself,
@@ -417,124 +429,192 @@ Yet to come
 
 ### Running SMOKE suite on the DirectRunner (local)
 
+The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`.
+Here we do it for maximum clarity.
+
+The direct runner does not have separate batch and streaming modes, but the
+Nexmark launch does.
+
+These parameters leave on many of the DirectRunner's extra safety checks so the
+SMOKE suite can make sure there is nothing broken in the Nexmark suite.
+
 Batch Mode:
 
-mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
-Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE 
--streaming=false --manageResources=false --monitorJobs=true 
--enforceEncodability=true --enforceImmutability=true"
+./gradlew :beam-sdks-java-nexmark:run \
+-Pnexmark.runner=":beam-runners-direct-java" \
+-Pnexmark.args="
+--runner=DirectRunner
+--streaming=false
+--suite=SMOKE
+--manageResources=false

[beam-site] branch mergebot updated (ef75570 -> 7cc5dfd)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from ef75570  This closes #427
 add 9193579  Prepare repository for deployment.
 new 523bbd0  Update Nexmark launch instructions for Gradle
 new 7cc5dfd  This closes #418

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html |  39 ++--
 src/documentation/sdks/nexmark.md| 270 +++
 2 files changed, 194 insertions(+), 115 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95341
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 04:55
Start Date: 26/Apr/18 04:55
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#issuecomment-384512907
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95341)
Time Spent: 10h  (was: 9h 50m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=95338&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95338
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 26/Apr/18 04:45
Start Date: 26/Apr/18 04:45
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #418: [BEAM-3995] Update 
Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#issuecomment-384511690
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95338)
Time Spent: 9h 50m  (was: 9h 40m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (8e8a304 -> 9193579)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 8e8a304  Prepare repository for deployment.
 add 5a1d8a0  Update contribution guide to reflect current practices.
 add 265088e  Further refinement.
 add ef75570  This closes #427
 new 9193579  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html | 39 
 src/contribute/contribution-guide.md | 37 +++---
 2 files changed, 37 insertions(+), 39 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9193579c22e1f4c01ec3d34fbec026def3280e47
Author: Mergebot 
AuthorDate: Wed Apr 25 21:44:16 2018 -0700

Prepare repository for deployment.
---
 content/contribute/contribution-guide/index.html | 39 
 1 file changed, 19 insertions(+), 20 deletions(-)

diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index 83653dc..2228bc1 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -242,7 +242,8 @@ repository.
 mailing list, which is archived
 https://lists.apache.org/list.html?d...@beam.apache.org";>here. 
Join by
 emailing
-mailto:dev-subscr...@beam.apache.org";>dev-subscr...@beam.apache.org.
+mailto:dev-subscr...@beam.apache.org";>dev-subscr...@beam.apache.org.
+Anyone is welcome to join, and we’d love to hear what you’re interested in.
 
 If interested, you can also join the other mailing lists.
 
@@ -250,10 +251,7 @@ emailing
 
 We use the Apache Software Foundation’s
 https://issues.apache.org/jira/browse/BEAM";>JIRA as an issue 
tracking and
-project management tool, as well as a way to communicate among a very diverse
-and distributed set of contributors. To be able to gather feedback, avoid
-frustration, and avoid duplicated efforts all Beam-related work should be
-tracked there.
+project management tool.
 
 If you do not already have an Apache JIRA account, sign up
 https://issues.apache.org/jira/";>here.
@@ -261,20 +259,19 @@ tracked there.
 If a quick
 https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22";>search
 doesn’t turn up an existing JIRA issue for the work you want to contribute,
-create it. Please discuss your idea with a committer or the https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel";>component
-lead
-in JIRA or, alternatively, on the developer mailing list.
+feel free to create one.
 
 If there’s an existing JIRA issue for your intended contribution, please
-comment about your intended work. Once the work is understood, a committer will
+comment about your intended work. A committer can
 assign the issue to you. (If you don’t have a JIRA role yet, you’ll be added to
-the “contributor” role.) If an issue is currently assigned, please check with
-the current assignee before reassigning.
+the “contributor” role.) If an issue is currently assigned, it is generally
+good practice to at least reach out the current assignee before 
reassigning.
 
-For moderate or large contributions, you should not start coding or writing 
a
-design document unless there is a corresponding JIRA issue assigned to you for
-that work. Simple changes, like fixing typos, do not require an associated
-issue.
+For moderate or large contributions, we recommend engaging on the developer
+mailing list before investing significant effort to gather feedback,
+avoid frustration, and prevent duplicated work.
+Simple changes, like fixing typos, do not nead JIRA entries
+and are welcome via direct pull requests.
 
 Online discussions
 
@@ -297,10 +294,12 @@ tracking system and/or the dev mailing list.
 to clearly scope and design non-trivial contributions with the Beam community
 before you start coding.
 
-Generally, the JIRA issue is the best place to gather relevant design docs,
-comments, or references. It’s great to explicitly include relevant stakeholders
-early in the conversation. For designs that may be generally interesting, we
-also encourage conversations on the developer’s mailing list.
+Generally, the developer mailing list is the best place to get help,
+solicit feedback, and communicate design docs.
+It’s great to explicitly include relevant stakeholders
+early in the conversation. JIRA is useful for keeping track of the current
+state of implementation and automatically gets updated for actions on pull
+requests that start with [BEAM-JIRA_ID].
 
 We suggest using https://docs.google.com/";>Google Docs for 
sharing designs
 that may benefit from diagrams or comments. Please remember to make the
@@ -495,7 +494,7 @@ $ python setup.py sdist
 Review
 
 Once the initial code is complete and the tests pass, it’s time to start the
-code review process. We review and discuss all code, no matter who authors it.
+code review process. All code is peer reviewed, no matter who authors it.
 It’s a great way to build community, since you can learn from other developers,
 and they become familiar with your contribution. It also builds a strong
 project by encouraging a high quality bar and keeping code consistent

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 03/03: This closes #427

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ef755705e78295b0a4f7d7973a3d11d27200cc21
Merge: 8e8a304 265088e
Author: Mergebot 
AuthorDate: Wed Apr 25 21:40:34 2018 -0700

This closes #427

 src/contribute/contribution-guide.md | 37 ++--
 1 file changed, 18 insertions(+), 19 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (75a17ae -> ef75570)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 75a17ae  This closes #428
 add 8e8a304  Prepare repository for deployment.
 new 5a1d8a0  Update contribution guide to reflect current practices.
 new 265088e  Further refinement.
 new ef75570  This closes #427

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html |  6 ++--
 src/contribute/contribution-guide.md | 37 
 2 files changed, 21 insertions(+), 22 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/03: Further refinement.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 265088ec44da4dceab24c9ce0b6570eda1984488
Author: Robert Bradshaw 
AuthorDate: Wed Apr 25 09:45:19 2018 -0700

Further refinement.
---
 src/contribute/contribution-guide.md | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 21cd434..3c16092 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -66,6 +66,7 @@ mailing list, which is archived
 [here](https://lists.apache.org/list.html?d...@beam.apache.org). Join by
 emailing
 [`dev-subscr...@beam.apache.org`](mailto:dev-subscr...@beam.apache.org).
+Anyone is welcome to join, and we'd love to hear what you're interested in.
 
 If interested, you can also join the other [mailing lists]({{ site.baseurl
 }}/get-started/support/).
@@ -74,9 +75,7 @@ If interested, you can also join the other [mailing lists]({{ 
site.baseurl
 
 We use the Apache Software Foundation's
 [JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue tracking and
-project management tool, as well as a way to communicate among a very diverse
-and distributed set of contributors. To avoid duplicated efforts,
-significant Beam-related work should be tracked there.
+project management tool.
 
 If you do not already have an Apache JIRA account, sign up
 [here](https://issues.apache.org/jira/).
@@ -84,9 +83,7 @@ If you do not already have an Apache JIRA account, sign up
 If a quick
 
[search](https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22)
 doesn’t turn up an existing JIRA issue for the work you want to contribute,
-feel free to create one. Please discuss your idea with a committer or the 
[component
-lead](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel)
-in JIRA or, alternatively, on the developer mailing list.
+feel free to create one.
 
 If there’s an existing JIRA issue for your intended contribution, please
 comment about your intended work. A committer can
@@ -303,7 +300,7 @@ $ python setup.py sdist > /dev/null && \
 ## Review
 
 Once the initial code is complete and the tests pass, it’s time to start the
-code review process. We review and discuss all code, no matter who authors it.
+code review process. All code is peer reviewed, no matter who authors it.
 It’s a great way to build community, since you can learn from other developers,
 and they become familiar with your contribution. It also builds a strong
 project by encouraging a high quality bar and keeping code consistent

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/03: Update contribution guide to reflect current practices.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 5a1d8a00422d84266b27f58812ca77f48849dc6c
Author: Robert Bradshaw 
AuthorDate: Tue Apr 24 18:18:02 2018 -0700

Update contribution guide to reflect current practices.

Specifically, our use of JIRA vs. the mailing list.

I also tried to be less prescriptive to lower the bar for people just 
getting started.
---
 src/contribute/contribution-guide.md | 32 +---
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 9b614dd..21cd434 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -75,9 +75,8 @@ If interested, you can also join the other [mailing lists]({{ 
site.baseurl
 We use the Apache Software Foundation's
 [JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue tracking and
 project management tool, as well as a way to communicate among a very diverse
-and distributed set of contributors. To be able to gather feedback, avoid
-frustration, and avoid duplicated efforts all Beam-related work should be
-tracked there.
+and distributed set of contributors. To avoid duplicated efforts,
+significant Beam-related work should be tracked there.
 
 If you do not already have an Apache JIRA account, sign up
 [here](https://issues.apache.org/jira/).
@@ -85,20 +84,21 @@ If you do not already have an Apache JIRA account, sign up
 If a quick
 
[search](https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22)
 doesn’t turn up an existing JIRA issue for the work you want to contribute,
-create it. Please discuss your idea with a committer or the [component
+feel free to create one. Please discuss your idea with a committer or the 
[component
 
lead](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel)
 in JIRA or, alternatively, on the developer mailing list.
 
 If there’s an existing JIRA issue for your intended contribution, please
-comment about your intended work. Once the work is understood, a committer will
+comment about your intended work. A committer can
 assign the issue to you. (If you don’t have a JIRA role yet, you’ll be added to
-the “contributor” role.) If an issue is currently assigned, please check with
-the current assignee before reassigning.
+the “contributor” role.) If an issue is currently assigned, it is generally
+good practice to at least reach out the current assignee before reassigning.
 
-For moderate or large contributions, you should not start coding or writing a
-design document unless there is a corresponding JIRA issue assigned to you for
-that work. Simple changes, like fixing typos, do not require an associated
-issue.
+For moderate or large contributions, we recommend engaging on the developer
+mailing list before investing significant effort to gather feedback,
+avoid frustration, and prevent duplicated work.
+Simple changes, like fixing typos, do not nead JIRA entries
+and are welcome via direct pull requests.
 
 ### Online discussions
 
@@ -121,10 +121,12 @@ To avoid potential frustration during the code review 
cycle, we encourage you
 to clearly scope and design non-trivial contributions with the Beam community
 before you start coding.
 
-Generally, the JIRA issue is the best place to gather relevant design docs,
-comments, or references. It’s great to explicitly include relevant stakeholders
-early in the conversation. For designs that may be generally interesting, we
-also encourage conversations on the developer’s mailing list.
+Generally, the developer mailing list is the best place to get help,
+solicit feedback, and communicate design docs.
+It’s great to explicitly include relevant stakeholders
+early in the conversation. JIRA is useful for keeping track of the current
+state of implementation and automatically gets updated for actions on pull
+requests that start with `[BEAM-JIRA_ID]`.
 
 We suggest using [Google Docs](https://docs.google.com/) for sharing designs
 that may benefit from diagrams or comments. Please remember to make the

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch asf-site updated (3acc115 -> 8e8a304)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 3acc115  Prepare repository for deployment.
 add e7046b4  Cleanup now that GitHub is not the mirror.
 add 75a17ae  This closes #428
 new 8e8a304  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html | 6 +++---
 src/contribute/contribution-guide.md | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8e8a304bb115886375ab6009feae4061f232dad8
Author: Mergebot 
AuthorDate: Wed Apr 25 21:36:12 2018 -0700

Prepare repository for deployment.
---
 content/contribute/contribution-guide/index.html | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index 55f2d9f..83653dc 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -342,8 +342,8 @@ contributions and do not require an ICLA in that case.
 
 Fork the repository on GitHub
 
-Go to the https://github.com/apache/beam/";>Beam GitHub mirror 
and fork the
-repository to your own private account. This will be your private workspace for
+Go to the https://github.com/apache/beam/";>Beam GitHub 
repository and fork the
+it to your own private account. This will be your private workspace for
 staging changes.
 
 Clone the repository locally
@@ -356,7 +356,7 @@ development.
 exchange SSH keys with GitHub by following https://help.github.com/articles/generating-an-ssh-key/";>their
 instructions.
 
-Clone Beam’s read-only GitHub mirror.
+Clone Beam’s GitHub repository.
 
 $ git clone 
https://github.com/apache/beam.git
 $ cd beam

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #428

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 75a17ae77233de2f572f1b5ded908ec02b18bfd7
Merge: 3acc115 e7046b4
Author: Mergebot 
AuthorDate: Wed Apr 25 21:33:31 2018 -0700

This closes #428

 src/contribute/contribution-guide.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (4196099 -> 75a17ae)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 4196099  This closes #428
 discard 91c3a33  Cleanup now that GitHub is not the mirror.
 new e7046b4  Cleanup now that GitHub is not the mirror.
 new 75a17ae  This closes #428

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (4196099)
\
 N -- N -- N   refs/heads/mergebot (75a17ae)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Cleanup now that GitHub is not the mirror.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e7046b465ae22f7a317b9b7d0f0fde48f8245da2
Author: Robert Bradshaw 
AuthorDate: Tue Apr 24 18:24:22 2018 -0700

Cleanup now that GitHub is not the mirror.
---
 src/contribute/contribution-guide.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 44ea14b..9b614dd 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -167,8 +167,8 @@ If you do not already have a personal GitHub account, sign 
up
 
  Fork the repository on GitHub
 
-Go to the [Beam GitHub mirror](https://github.com/apache/beam/) and fork the
-repository to your own private account. This will be your private workspace for
+Go to the [Beam GitHub repository](https://github.com/apache/beam/) and fork 
the
+it to your own private account. This will be your private workspace for
 staging changes.
 
  Clone the repository locally
@@ -181,7 +181,7 @@ We assume you are using SSH-based authentication with 
GitHub. If necessary,
 exchange SSH keys with GitHub by following [their
 instructions](https://help.github.com/articles/generating-an-ssh-key/).
 
-Clone Beam’s read-only GitHub mirror.
+Clone Beam’s GitHub repository.
 
 $ git clone https://github.com/apache/beam.git
 $ cd beam

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-3925) Allow ValueProvider for KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=95334&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95334
 ]

ASF GitHub Bot logged work on BEAM-3925:


Author: ASF GitHub Bot
Created on: 26/Apr/18 04:22
Start Date: 26/Apr/18 04:22
Worklog Time Spent: 10m 
  Work Description: pupamanyu commented on a change in pull request #5141: 
[BEAM-3925] Allow ValueProvider for KafkaIO so that we can create Beam 
Templates using KafkaIO
URL: https://github.com/apache/beam/pull/5141#discussion_r183844484
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -359,20 +386,129 @@
  * of how the partitions are distributed among the splits.
  */
 public Read withTopicPartitions(List 
topicPartitions) {
-  checkState(getTopics().isEmpty(), "Only topics or topicPartitions can be 
set, not both");
-  return 
toBuilder().setTopicPartitions(ImmutableList.copyOf(topicPartitions)).build();
+  return withTopicPartitions(
+  
ValueProvider.StaticValueProvider.of(Joiner.on(',').join(topicPartitions)));
+}
+
+/**
+ * Sets a list of partitions to read from. Partitions are provided as a 
comma separated list of
+ * Strings in the format: topic1-partition1,topic1-partition2...
+ * This allows reading only a subset of partitions for one or more topics 
when (if ever) needed.
+ *
+ * See {@link KafkaUnboundedSource#split(int, PipelineOptions)} for 
description
+ * of how the partitions are distributed among the splits.
+ */
+public Read withTopicPartitions(String topicPartitions) {
+  return 
withTopicPartitions(ValueProvider.StaticValueProvider.of(topicPartitions));
+}
+
+/**
+ * Like above but with a {@link ValueProvider ValueProvider}.
+ */
+public Read withTopicPartitions(ValueProvider 
topicPartitions) {
+  checkState(getTopics().get().isEmpty(),
+  "Only topics or topicPartitions can be set, not both");
+  return toBuilder().setTopicPartitions(ValueProvider
+  .NestedValueProvider.of(topicPartitions, new 
TopicPartitionTranslator())).build();
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link List 
List}.
+ */
+private static class TopicTranslator implements 
SerializableFunction> {
+  @Override
+  public List apply(String topics) {
+return ImmutableList.copyOf(
+
Splitter.on(',').trimResults().omitEmptyStrings().splitToList(topics));
+  }
+
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link List 
List}.
+ */
+private static class TopicPartitionTranslator
+implements SerializableFunction> {
+  @Override
+  public List apply(String topicPartitions) {
+List topicPartitionList = new ArrayList<>();
+for (String topicPartition: 
Splitter.on(',').trimResults().omitEmptyStrings()
+  .splitToList(topicPartitions)) {
+  topicPartitionList
+  .add(new 
TopicPartition(Splitter.on('-').splitToList(topicPartition).get(0),
+  
Integer.parseInt(Splitter.on('-').splitToList(topicPartition).get(1;
+}
+  return ImmutableList.copyOf(topicPartitionList);
+  }
+}
+
+/**
+ * Sets a Kafka {@link Deserializer Deserializer} for 
interpreting key bytes read.
+ * This uses the {@link String} provided to set the Deserializer
+ */
+public Read withKeyDeserializer(String keyDeserializer) {
+  return 
withKeyDeserializer(ValueProvider.StaticValueProvider.of(keyDeserializer));
+}
+
+/**
+ * Like above but with a {@link ValueProvider ValueProvider}.
+ */
+public Read withKeyDeserializer(ValueProvider 
keyDeserializer) {
+  return toBuilder().setKeyDeserializer(ValueProvider
+  .NestedValueProvider.of(keyDeserializer, new 
KeyDeserializerTranslator())).build();
 }
 
 /**
- * Sets a Kafka {@link Deserializer} to interpret key bytes read from 
Kafka.
+ * Sets a Kafka {@link Deserializer Deserializer} to interpret 
key bytes read.
  *
  * In addition, Beam also needs a {@link Coder} to serialize and 
deserialize key objects at
  * runtime. KafkaIO tries to infer a coder for the key based on the {@link 
Deserializer} class,
  * however in case that fails, you can use {@link 
#withKeyDeserializerAndCoder(Class, Coder)} to
  * provide the key coder explicitly.
  */
 public Read withKeyDeserializer(Class> 
keyDeserializer) {
-  return toBuilder().setKeyDeserializer(keyDeserializer).build();
+  return toBuilder().setKeyDeserializer(
+  ValueProvider.StaticValueProvider.of(keyDeserializer)).build();
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link Deserializer 

[jira] [Work logged] (BEAM-3925) Allow ValueProvider for KafkaIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=95332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95332
 ]

ASF GitHub Bot logged work on BEAM-3925:


Author: ASF GitHub Bot
Created on: 26/Apr/18 04:21
Start Date: 26/Apr/18 04:21
Worklog Time Spent: 10m 
  Work Description: pupamanyu commented on a change in pull request #5141: 
[BEAM-3925] Allow ValueProvider for KafkaIO so that we can create Beam 
Templates using KafkaIO
URL: https://github.com/apache/beam/pull/5141#discussion_r183844484
 
 

 ##
 File path: 
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
 ##
 @@ -359,20 +386,129 @@
  * of how the partitions are distributed among the splits.
  */
 public Read withTopicPartitions(List 
topicPartitions) {
-  checkState(getTopics().isEmpty(), "Only topics or topicPartitions can be 
set, not both");
-  return 
toBuilder().setTopicPartitions(ImmutableList.copyOf(topicPartitions)).build();
+  return withTopicPartitions(
+  
ValueProvider.StaticValueProvider.of(Joiner.on(',').join(topicPartitions)));
+}
+
+/**
+ * Sets a list of partitions to read from. Partitions are provided as a 
comma separated list of
+ * Strings in the format: topic1-partition1,topic1-partition2...
+ * This allows reading only a subset of partitions for one or more topics 
when (if ever) needed.
+ *
+ * See {@link KafkaUnboundedSource#split(int, PipelineOptions)} for 
description
+ * of how the partitions are distributed among the splits.
+ */
+public Read withTopicPartitions(String topicPartitions) {
+  return 
withTopicPartitions(ValueProvider.StaticValueProvider.of(topicPartitions));
+}
+
+/**
+ * Like above but with a {@link ValueProvider ValueProvider}.
+ */
+public Read withTopicPartitions(ValueProvider 
topicPartitions) {
+  checkState(getTopics().get().isEmpty(),
+  "Only topics or topicPartitions can be set, not both");
+  return toBuilder().setTopicPartitions(ValueProvider
+  .NestedValueProvider.of(topicPartitions, new 
TopicPartitionTranslator())).build();
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link List 
List}.
+ */
+private static class TopicTranslator implements 
SerializableFunction> {
+  @Override
+  public List apply(String topics) {
+return ImmutableList.copyOf(
+
Splitter.on(',').trimResults().omitEmptyStrings().splitToList(topics));
+  }
+
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link List 
List}.
+ */
+private static class TopicPartitionTranslator
+implements SerializableFunction> {
+  @Override
+  public List apply(String topicPartitions) {
+List topicPartitionList = new ArrayList<>();
+for (String topicPartition: 
Splitter.on(',').trimResults().omitEmptyStrings()
+  .splitToList(topicPartitions)) {
+  topicPartitionList
+  .add(new 
TopicPartition(Splitter.on('-').splitToList(topicPartition).get(0),
+  
Integer.parseInt(Splitter.on('-').splitToList(topicPartition).get(1;
+}
+  return ImmutableList.copyOf(topicPartitionList);
+  }
+}
+
+/**
+ * Sets a Kafka {@link Deserializer Deserializer} for 
interpreting key bytes read.
+ * This uses the {@link String} provided to set the Deserializer
+ */
+public Read withKeyDeserializer(String keyDeserializer) {
+  return 
withKeyDeserializer(ValueProvider.StaticValueProvider.of(keyDeserializer));
+}
+
+/**
+ * Like above but with a {@link ValueProvider ValueProvider}.
+ */
+public Read withKeyDeserializer(ValueProvider 
keyDeserializer) {
+  return toBuilder().setKeyDeserializer(ValueProvider
+  .NestedValueProvider.of(keyDeserializer, new 
KeyDeserializerTranslator())).build();
 }
 
 /**
- * Sets a Kafka {@link Deserializer} to interpret key bytes read from 
Kafka.
+ * Sets a Kafka {@link Deserializer Deserializer} to interpret 
key bytes read.
  *
  * In addition, Beam also needs a {@link Coder} to serialize and 
deserialize key objects at
  * runtime. KafkaIO tries to infer a coder for the key based on the {@link 
Deserializer} class,
  * however in case that fails, you can use {@link 
#withKeyDeserializerAndCoder(Class, Coder)} to
  * provide the key coder explicitly.
  */
 public Read withKeyDeserializer(Class> 
keyDeserializer) {
-  return toBuilder().setKeyDeserializer(keyDeserializer).build();
+  return toBuilder().setKeyDeserializer(
+  ValueProvider.StaticValueProvider.of(keyDeserializer)).build();
+}
+
+/**
+ * Used to build a {@link ValueProvider} for {@link Deserializer 

[beam-site] branch mergebot updated (da98d3c -> 4196099)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from da98d3c  This closes #426
 add 3acc115  Prepare repository for deployment.
 new 91c3a33  Cleanup now that GitHub is not the mirror.
 new 4196099  This closes #428

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html | 34 +++-
 src/contribute/contribution-guide.md |  6 ++---
 2 files changed, 30 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #428

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4196099f38f79914c8455f9fc51acffb2f2dec7f
Merge: 3acc115 91c3a33
Author: Mergebot 
AuthorDate: Wed Apr 25 21:09:41 2018 -0700

This closes #428

 src/contribute/contribution-guide.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Cleanup now that GitHub is not the mirror.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 91c3a33a5468960875d979d54157c1e6acd7906f
Author: Robert Bradshaw 
AuthorDate: Tue Apr 24 18:24:22 2018 -0700

Cleanup now that GitHub is not the mirror.
---
 src/contribute/contribution-guide.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 44ea14b..9b614dd 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -167,8 +167,8 @@ If you do not already have a personal GitHub account, sign 
up
 
  Fork the repository on GitHub
 
-Go to the [Beam GitHub mirror](https://github.com/apache/beam/) and fork the
-repository to your own private account. This will be your private workspace for
+Go to the [Beam GitHub repository](https://github.com/apache/beam/) and fork 
the
+it to your own private account. This will be your private workspace for
 staging changes.
 
  Clone the repository locally
@@ -181,7 +181,7 @@ We assume you are using SSH-based authentication with 
GitHub. If necessary,
 exchange SSH keys with GitHub by following [their
 instructions](https://help.github.com/articles/generating-an-ssh-key/).
 
-Clone Beam’s read-only GitHub mirror.
+Clone Beam’s GitHub repository.
 
 $ git clone https://github.com/apache/beam.git
 $ cd beam

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #194

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add OutputDeduplicator

[tgroh] Update ExecutableStageMatcher to accept Matchers

[tgroh] Use OutputDeduplicator in GreedyPipelineFuser

[tgroh] fixup! Add OutputDeduplicator

--
[...truncated 18.03 MB...]
INFO: Uploading 
/home/jenkins/.m2/repository/org/apache/httpcomponents/httpclient/4.0.1/httpclient-4.0.1.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/httpclient-4.0.1-nKmHdIYBAcBsqQEO_WIkoQ.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/protobuf-java-3.2.0-fh30Gescj5k_IhxJOGxriw.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading /home/jenkins/.m2/repository/org/tukaani/xz/1.5/xz-1.5.jar 
to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/xz-1.5-UQUOWVswjErsisMU9m4YvA.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/error_prone_annotations-2.0.15-npnuyJrjs2dF75GdvUBIvQ.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.gradle/caches/modules-2/files-2.1/io.grpc/grpc-context/1.9.0/28b0836f48c9705abf73829bbc536dba29a1329a/grpc-context-1.9.0.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/grpc-context-1.9.0-atYmq4r_lax8i7-0BRHcCg.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/json/json/20160810/json-20160810.jar to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/json-20160810-L3-Jnwdm5lAXdEpMT8FNRg.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/jackson-mapper-asl-1.9.13-F1D5wzk1L8S3KNYbVxcWEw.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/mockito/mockito-core/1.9.5/mockito-core-1.9.5.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/mockito-core-1.9.5-b3PPBKVutgqqmWUG58EPxw.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/netty-handler-proxy-4.1.8.Final-Zey48Fj4mlWtgpwIc7Osgg.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-codec-socks/4.1.8.Final/netty-codec-socks-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/netty-codec-socks-4.1.8.Final-dfbCpCpgkEl1iwscd6OVeQ.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-codec/4.1.8.Final/netty-codec-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/netty-codec-4.1.8.Final-Og5DJ4MB-fn1oSzjKqVU_w.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/apache/commons/commons-compress/1.16.1/commons-compress-1.16.1.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/output/results/staging/commons-compress-1.16.1-NAljjWtr0jBC7qxc2X4lbA.jar
Apr 26, 2018 2:56:06 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0426025604-3c8f0099/outpu

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #192

[Apache Jenkins Server]

See 


Changes:

[swegner] Upgrade Gradle to 4.7

--
[...truncated 19.83 MB...]
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.072Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.106Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.144Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.178Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.222Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.257Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.285Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.323Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.352Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.386Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Apr 26, 2018 2:52:00 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:51:56.420Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSi

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #193

[Apache Jenkins Server]

See 


Changes:

[swegner] Upgrade Gradle protobuf plugin to latest.

--
[...truncated 17.98 MB...]
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.160Z: Fusing unzipped copy of 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous), through 
flatten s18-u63, into producer 
PAssert$3/CreateActual/FilterActuals/Window.Assign
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.191Z: Fusing consumer 
PAssert$3/CreateActual/GatherPanes/Reify.Window/ParDo(Anonymous) into 
PAssert$3/CreateActual/FilterActuals/Window.Assign
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.214Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.242Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.275Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.294Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.358Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.394Z: Fusing consumer 
PAssert$3/CreateActual/RewindowActuals/Window.Assign into 
PAssert$3/CreateActual/Flatten.Iterables/FlattenIterables/FlatMap
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.423Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.458Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Apr 26, 2018 2:34:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-04-26T02:34:35.485Z: Fusing consumer 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into 
PAssert$3/CreateActual/View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Apr 26,

[jira] [Work logged] (BEAM-3914) 'Unzip' flattens before performing fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3914?focusedWorklogId=95324&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95324
 ]

ASF GitHub Bot logged work on BEAM-3914:


Author: ASF GitHub Bot
Created on: 26/Apr/18 02:04
Start Date: 26/Apr/18 02:04
Worklog Time Spent: 10m 
  Work Description: tgroh closed pull request #4977: [BEAM-3914] 
Deduplicate Unzipped Flattens after Pipeline Fusion
URL: https://github.com/apache/beam/pull/4977
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
index 68da5c3961b..ddc03355a90 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
@@ -98,28 +98,16 @@ static FusedPipeline of(
   private Map getEnvironmentExecutedTransforms() {
 Map topLevelTransforms = new HashMap<>();
 for (ExecutableStage stage : getFusedStages()) {
+  String baseName =
+  String.format(
+  "%s/%s",
+  stage.getInputPCollection().getPCollection().getUniqueName(),
+  stage.getEnvironment().getUrl());
+  Set usedNames =
+  Sets.union(topLevelTransforms.keySet(), 
getComponents().getTransformsMap().keySet());
   topLevelTransforms.put(
-  generateStageId(
-  stage,
-  Sets.union(getComponents().getTransformsMap().keySet(), 
topLevelTransforms.keySet())),
-  stage.toPTransform());
+  SyntheticNodes.uniqueId(baseName, usedNames::contains), 
stage.toPTransform());
 }
 return topLevelTransforms;
   }
-
-  private String generateStageId(ExecutableStage stage, Set 
existingIds) {
-int i = 0;
-String name;
-do {
-  // Instead this could include the name of the root transforms
-  name =
-  String.format(
-  "%s/%s.%s",
-  stage.getInputPCollection().getPCollection().getUniqueName(),
-  stage.getEnvironment().getUrl(),
-  i);
-  i++;
-} while (existingIds.contains(name));
-return name;
-  }
 }
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
index abbba2a91a9..aaa44b67c00 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
@@ -27,6 +27,7 @@
 import com.google.common.collect.Multimap;
 import com.google.common.collect.Sets;
 import java.util.ArrayDeque;
+import java.util.Collection;
 import java.util.Comparator;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -42,6 +43,7 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
 import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
 import org.apache.beam.runners.core.construction.PTransformTranslation;
+import 
org.apache.beam.runners.core.construction.graph.OutputDeduplicator.DeduplicationResult;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
 import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
 import org.slf4j.Logger;
@@ -54,19 +56,19 @@
   private static final Logger LOG = 
LoggerFactory.getLogger(GreedyPipelineFuser.class);
 
   private final QueryablePipeline pipeline;
-  private final Map 
consumedCollectionsAndTransforms =
-  new HashMap<>();
-  private final Set unfusedTransforms = new LinkedHashSet<>();
-  private final Set stages = new LinkedHashSet<>();
+  private final FusedPipeline fusedPipeline;
 
   private GreedyPipelineFuser(Pipeline p) {
 this.pipeline = QueryablePipeline.forPrimitivesIn(p.getComponents());
+Set unfusedRootNodes = new LinkedHashSet<>();
 NavigableSet rootConsumers = new TreeSet<>();
 for (PTransformNode pTransformNode : pipeline.getRootTransforms()) {
   // This will usually be a single node, the downstream of an Impulse, but 
may be of any size
-  rootConsumers.addAll(getRootEnvTransforms(pTransformNode));
+  DescendantConsumers descendants = getRootConsumers(pTransformNode);
+  unfusedRootNodes.addAll(descen

[jira] [Work logged] (BEAM-3914) 'Unzip' flattens before performing fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3914?focusedWorklogId=95323&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95323
 ]

ASF GitHub Bot logged work on BEAM-3914:


Author: ASF GitHub Bot
Created on: 26/Apr/18 02:04
Start Date: 26/Apr/18 02:04
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4977: 
[BEAM-3914] Deduplicate Unzipped Flattens after Pipeline Fusion
URL: https://github.com/apache/beam/pull/4977#discussion_r184253375
 
 

 ##
 File path: 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
 ##
 @@ -114,47 +125,66 @@ public void unchangedWithNoDuplicates() {
   public void duplicateOverStages() {
 /* When multiple stages and a runner-executed transform produce a 
PCollection, all should be
  * replaced with synthetic flattens.
- * S -> A; T -> A becomes S -> A'; T -> A''; A', A'' -> Flatten -> A
+ * original graph:
+ * --> one -> .out \
+ * red -> .out |-> shared -> .out -> blue -> .out
+ * --> two -> .out /
+ *
+ * fused graph:
+ * --> [one -> .out -> shared ->] .out
+ * red -> .out |   (shared.out) -> blue -> 
.out
+ * --> [two -> .out -> shared ->] .out
+ *
+ * deduplicated graph:
+ * --> [one -> .out -> shared ->] .out:0 \
+ * red -> .out |  -> shared -> .out -> 
blue ->.out
+ * --> [two -> .out -> shared ->] .out:1 /
  */
-PTransform one =
-PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"one.out").build();
+PCollection redOut = 
PCollection.newBuilder().setUniqueName("red.out").build();
+PTransform red = PTransform.newBuilder().putOutputs("out", 
redOut.getUniqueName()).build();
 PCollection oneOut = 
PCollection.newBuilder().setUniqueName("one.out").build();
-PTransform two =
-PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"two.out").build();
+PTransform one =
+PTransform.newBuilder()
+.putInputs("in", redOut.getUniqueName())
+.putOutputs("out", oneOut.getUniqueName())
+.build();
 PCollection twoOut = 
PCollection.newBuilder().setUniqueName("two.out").build();
+PTransform two =
+PTransform.newBuilder()
+.putInputs("in", redOut.getUniqueName())
+.putOutputs("out", twoOut.getUniqueName())
+.build();
 PCollection sharedOut = 
PCollection.newBuilder().setUniqueName("shared.out").build();
 PTransform shared =
 PTransform.newBuilder()
-.putInputs("one", "one.out")
-.putInputs("two", "two.out")
+.putInputs("one", oneOut.getUniqueName())
+.putInputs("two", twoOut.getUniqueName())
 .putOutputs("shared", sharedOut.getUniqueName())
 .build();
-PTransform red = PTransform.newBuilder().putOutputs("out", 
"red.out").build();
-PCollection redOut = 
PCollection.newBuilder().setUniqueName("red.out").build();
+PCollection blueOut = 
PCollection.newBuilder().setUniqueName("blue.out").build();
 PTransform blue =
 PTransform.newBuilder()
 .putInputs("in", sharedOut.getUniqueName())
-.putOutputs("out", "blue.out")
+.putOutputs("out", blueOut.getUniqueName())
 .build();
-PCollection blueOut = 
PCollection.newBuilder().setUniqueName("blue.out").build();
 RunnerApi.Components components =
 Components.newBuilder()
 .putTransforms("one", one)
-.putPcollections("one.out", oneOut)
+.putPcollections(oneOut.getUniqueName(), oneOut)
 
 Review comment:
   It'd be nice, but `unique_name` is a custom field and we're just 
manipulating the protos directly here.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95323)
Time Spent: 3h 50m  (was: 3h 40m)

> 'Unzip' flattens before performing fusion
> -
>
> Key: BEAM-3914
> URL: https://issues.apache.org/jira/browse/BEAM-3914
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 50m
>  Remaining Estimate:

[beam] 01/01: Merge pull request #5223: Upgrade Gradle protobuf plugin to latest.

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2a7d7b1fe1828ee190b3d4408270b4b80e30df2d
Merge: d7efddb b815e7b
Author: Kenn Knowles 
AuthorDate: Wed Apr 25 18:46:22 2018 -0700

Merge pull request #5223: Upgrade Gradle protobuf plugin to latest.

 build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --cc build.gradle
index b8119d4,f3759c4..591c429
--- a/build.gradle
+++ b/build.gradle
@@@ -49,10 -49,9 +49,10 @@@ buildscript 
  maven { url "http://repo.spring.io/plugins-release"; }
}
dependencies {
 +classpath 'net.researchgate:gradle-release:2.6.0' 
  // Enable gradle-based release management
  classpath "com.gradle:build-scan-plugin:1.13.1"   
  // Enable publishing build scans
  classpath "net.ltgt.gradle:gradle-apt-plugin:0.13"
  // Enable a Java annotation processor
- classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.1"  
  // Enable proto code generation
+ classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.5"  
  // Enable proto code generation
  classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE"
  // Enable provided and optional configurations
  classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
  // Enable Apache license enforcement
  classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"   
  // Enable Avro code generation

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (d7efddb -> 2a7d7b1)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d7efddb  Merge pull request #5217: Upgrade Gradle to 4.7
 add b815e7b  Upgrade Gradle protobuf plugin to latest.
 new 2a7d7b1  Merge pull request #5223: Upgrade Gradle protobuf plugin to 
latest.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5217: Upgrade Gradle to 4.7

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d7efddbf78d2ab9b6cd8f220d7120167648778c8
Merge: bbabb42 a2e4531
Author: Kenn Knowles 
AuthorDate: Wed Apr 25 18:36:41 2018 -0700

Merge pull request #5217: Upgrade Gradle to 4.7

 gradle/wrapper/gradle-wrapper.properties | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (bbabb42 -> d7efddb)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bbabb42  Merge pull request #4672: Moves standard URNs into .proto 
files
 add a2e4531  Upgrade Gradle to 4.7
 new d7efddb  Merge pull request #5217: Upgrade Gradle to 4.7

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 gradle/wrapper/gradle-wrapper.properties | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-3914) 'Unzip' flattens before performing fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3914?focusedWorklogId=95317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95317
 ]

ASF GitHub Bot logged work on BEAM-3914:


Author: ASF GitHub Bot
Created on: 26/Apr/18 01:11
Start Date: 26/Apr/18 01:11
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #4977: 
[BEAM-3914] Deduplicate Unzipped Flattens after Pipeline Fusion
URL: https://github.com/apache/beam/pull/4977#discussion_r184247525
 
 

 ##
 File path: 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
 ##
 @@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction.graph;
+
+import static com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasEntry;
+import static org.hamcrest.Matchers.hasItems;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ImmutableList;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import 
org.apache.beam.runners.core.construction.graph.OutputDeduplicator.DeduplicationResult;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link OutputDeduplicator}. */
+@RunWith(JUnit4.class)
+public class OutputDeduplicatorTest {
+  @Test
+  public void unchangedWithNoDuplicates() {
+/* When all the PCollections are produced by only one transform or stage, 
the result should be
+ * empty//identical to the input.
+ */
+PTransform one =
+PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"one.out").build();
+PCollection oneOut = 
PCollection.newBuilder().setUniqueName("one.out").build();
+PTransform two =
+PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"two.out").build();
+PCollection twoOut = 
PCollection.newBuilder().setUniqueName("two.out").build();
+PTransform red = PTransform.newBuilder().putOutputs("out", 
"red.out").build();
+PCollection redOut = 
PCollection.newBuilder().setUniqueName("red.out").build();
+PTransform blue =
+PTransform.newBuilder()
+.putInputs("one", "one.out")
+.putInputs("two", "two.out")
+.putOutputs("out", "blue.out")
+.build();
+PCollection blueOut = 
PCollection.newBuilder().setUniqueName("blue.out").build();
+RunnerApi.Components components =
+Components.newBuilder()
+.putTransforms("one", one)
+.putPcollections("one.out", oneOut)
+.putTransforms("two", two)
+.putPcollections("two.out", twoOut)
+.putTransforms("red", red)
+.putPcollections("red.out", redOut)
+.putTransforms("blue", blue)
+.putPcollections("blue.out", blueOut)
+.build();
+ExecutableStage oneStage =
+ImmutableExecutableStage.of(
+components,
+Environment.getDefaultInstance(),
+PipelineNode.pCollection("red.out", redOut),
+ImmutableList.of(),
+ImmutableList.of(PipelineNode.pTransform("one", one)),
+ImmutableList.of(PipelineNode.pCollection("one.out", oneOut)));
+ExecutableStage twoStage 

[jira] [Work logged] (BEAM-3914) 'Unzip' flattens before performing fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3914?focusedWorklogId=95319&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95319
 ]

ASF GitHub Bot logged work on BEAM-3914:


Author: ASF GitHub Bot
Created on: 26/Apr/18 01:11
Start Date: 26/Apr/18 01:11
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #4977: 
[BEAM-3914] Deduplicate Unzipped Flattens after Pipeline Fusion
URL: https://github.com/apache/beam/pull/4977#discussion_r184247459
 
 

 ##
 File path: 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
 ##
 @@ -114,47 +125,66 @@ public void unchangedWithNoDuplicates() {
   public void duplicateOverStages() {
 /* When multiple stages and a runner-executed transform produce a 
PCollection, all should be
  * replaced with synthetic flattens.
- * S -> A; T -> A becomes S -> A'; T -> A''; A', A'' -> Flatten -> A
+ * original graph:
+ * --> one -> .out \
+ * red -> .out |-> shared -> .out -> blue -> .out
+ * --> two -> .out /
+ *
+ * fused graph:
+ * --> [one -> .out -> shared ->] .out
+ * red -> .out |   (shared.out) -> blue -> 
.out
+ * --> [two -> .out -> shared ->] .out
+ *
+ * deduplicated graph:
+ * --> [one -> .out -> shared ->] .out:0 \
+ * red -> .out |  -> shared -> .out -> 
blue ->.out
+ * --> [two -> .out -> shared ->] .out:1 /
  */
-PTransform one =
-PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"one.out").build();
+PCollection redOut = 
PCollection.newBuilder().setUniqueName("red.out").build();
+PTransform red = PTransform.newBuilder().putOutputs("out", 
redOut.getUniqueName()).build();
 PCollection oneOut = 
PCollection.newBuilder().setUniqueName("one.out").build();
-PTransform two =
-PTransform.newBuilder().putInputs("in", "red.out").putOutputs("out", 
"two.out").build();
+PTransform one =
+PTransform.newBuilder()
+.putInputs("in", redOut.getUniqueName())
+.putOutputs("out", oneOut.getUniqueName())
+.build();
 PCollection twoOut = 
PCollection.newBuilder().setUniqueName("two.out").build();
+PTransform two =
+PTransform.newBuilder()
+.putInputs("in", redOut.getUniqueName())
+.putOutputs("out", twoOut.getUniqueName())
+.build();
 PCollection sharedOut = 
PCollection.newBuilder().setUniqueName("shared.out").build();
 PTransform shared =
 PTransform.newBuilder()
-.putInputs("one", "one.out")
-.putInputs("two", "two.out")
+.putInputs("one", oneOut.getUniqueName())
+.putInputs("two", twoOut.getUniqueName())
 .putOutputs("shared", sharedOut.getUniqueName())
 .build();
-PTransform red = PTransform.newBuilder().putOutputs("out", 
"red.out").build();
-PCollection redOut = 
PCollection.newBuilder().setUniqueName("red.out").build();
+PCollection blueOut = 
PCollection.newBuilder().setUniqueName("blue.out").build();
 PTransform blue =
 PTransform.newBuilder()
 .putInputs("in", sharedOut.getUniqueName())
-.putOutputs("out", "blue.out")
+.putOutputs("out", blueOut.getUniqueName())
 .build();
-PCollection blueOut = 
PCollection.newBuilder().setUniqueName("blue.out").build();
 RunnerApi.Components components =
 Components.newBuilder()
 .putTransforms("one", one)
-.putPcollections("one.out", oneOut)
+.putPcollections(oneOut.getUniqueName(), oneOut)
 
 Review comment:
   Feels like there should be a one-argument overload. (Or is there any reason 
for the names to not agree?)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95319)
Time Spent: 3h 40m  (was: 3.5h)

> 'Unzip' flattens before performing fusion
> -
>
> Key: BEAM-3914
> URL: https://issues.apache.org/jira/browse/BEAM-3914
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>

[jira] [Work logged] (BEAM-3914) 'Unzip' flattens before performing fusion

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3914?focusedWorklogId=95318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95318
 ]

ASF GitHub Bot logged work on BEAM-3914:


Author: ASF GitHub Bot
Created on: 26/Apr/18 01:11
Start Date: 26/Apr/18 01:11
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #4977: 
[BEAM-3914] Deduplicate Unzipped Flattens after Pipeline Fusion
URL: https://github.com/apache/beam/pull/4977#discussion_r184247179
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
 ##
 @@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction.graph;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+
+/**
+ * Utilities to insert synthetic {@link PCollectionNode PCollections} for 
{@link PCollection
+ * PCollections} which are produced by multiple independently executable 
stages.
+ */
+class OutputDeduplicator {
+
+  /**
+   * Ensure that no {@link PCollection} output by any of the {@code stages} or 
{@code
+   * unfusedTransforms} is produced by more than one of those stages or 
transforms.
+   *
+   * For each {@link PCollection} output by multiple stages and/or 
transforms, each producer is
+   * rewritten to produce a partial {@link PCollection}, which are then 
flattened together via an
+   * introduced Flatten node which produces the original output.
+   */
+  static DeduplicationResult ensureSingleProducer(
+  QueryablePipeline pipeline,
+  Collection stages,
+  Collection unfusedTransforms) {
+RunnerApi.Components.Builder unzippedComponents = 
pipeline.getComponents().toBuilder();
+
+Multimap pcollectionProducers =
+getProducers(pipeline, stages, unfusedTransforms);
+Multimap requiresNewOutput = 
HashMultimap.create();
+// Create a synthetic PCollection for each of these nodes. The transforms 
in the runner
+// portion of the graph that creates them should be replaced in the result 
components. The
+// ExecutableStage must also be rewritten to have updated outputs and 
transforms.
+for (Map.Entry> 
collectionProducer :
+pcollectionProducers.asMap().entrySet()) {
+  if (collectionProducer.getValue().size() > 1) {
+for (StageOrTransform producer : collectionProducer.getValue()) {
+  requiresNewOutput.put(producer, collectionProducer.getKey());
+}
+  }
+}
+
+Map updatedStages = new 
LinkedHashMap<>();
+Map updatedTransforms = new LinkedHashMap<>();
+Multimap originalToPartial = 
HashMultimap.create();
+for (Map.Entry> 
deduplicationTargets :
+requiresNewOutput.asMap().entrySet()) {
+  if (deduplicationTargets.getKey().getStage() != null) {
+StageDeduplication deduplication =
+deduplicatePCollections(
+deduplicationTargets.getKey().getStage(),
+deduplicationTargets.getValue(),
+unzippedComponents::containsPcollections);
+ 

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=95314&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95314
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 26/Apr/18 00:47
Start Date: 26/Apr/18 00:47
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r184205121
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,601 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.common.collect.BiMap;
+import com.google.common.collect.ImmutableBiMap;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.CoderTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
+import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
+import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
+import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api

[jira] [Work logged] (BEAM-3972) Flink runner translates batch pipelines directly by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3972?focusedWorklogId=95315&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95315
 ]

ASF GitHub Bot logged work on BEAM-3972:


Author: ASF GitHub Bot
Created on: 26/Apr/18 00:47
Start Date: 26/Apr/18 00:47
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5226: 
[BEAM-3972] Translate portable batch pipelines by proto
URL: https://github.com/apache/beam/pull/5226#discussion_r184221413
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
 ##
 @@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import java.util.List;
+import org.apache.flink.api.java.CollectionEnvironment;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.streaming.api.TimeCharacteristic;
+import 
org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Utilities for Flink execution environments. */
+public class FlinkExecutionEnvironments {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(FlinkExecutionEnvironments.class);
+
+  /**
+   * If the submitted job is a batch processing job, this method creates the 
adequate Flink {@link
+   * org.apache.flink.api.java.ExecutionEnvironment} depending on the 
user-specified options.
+   */
+  public static ExecutionEnvironment 
createBatchExecutionEnvironment(FlinkPipelineOptions options) {
 
 Review comment:
   Instead of using static methods, shall we use factory to create the 
environments. 
   Static methods are hard to swap and might create challenge while testing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95315)
Time Spent: 1h 40m  (was: 1.5h)

> Flink runner translates batch pipelines directly by proto
> -
>
> Key: BEAM-3972
> URL: https://issues.apache.org/jira/browse/BEAM-3972
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The non-portable runner uses reydrated pipelines which lack necessary 
> information. The portable Flink runner needs to translate pipelines directly 
> by proto in order to wire components into individual executable stages 
> correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #191

[Apache Jenkins Server]

See 


--
[...truncated 19.45 MB...]
org.apache.beam.examples.cookbook.FilterExamplesTest > 
testFilterSingleMonthDataFn STANDARD_ERROR
Apr 26, 2018 12:40:07 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
Apr 26, 2018 12:40:07 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
Apr 26, 2018 12:40:07 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
Apr 26, 2018 12:40:07 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
Apr 26, 2018 12:40:08 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit2685520963253068630/junit6928613568339758149.tmp matched 1 files with 
total size 54
Apr 26, 2018 12:40:08 AM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit2685520963253068630/junit6928613568339758149.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
Apr 26, 2018 12:40:08 AM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo703898095940123218.sh 
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo703898095940123218.sh to 
/tmp/test-Echoo3635545603389323111/test-Echo703898095940123218.sh 
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain2330251932287909884.sh 
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
Apr 26, 2018 12:40:09 AM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain2330251932287909884.sh to 
/tmp/test-Echoo3635545603389323111/test-EchoAgain2330251932287909884.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT THIS IS A PARSE ERROR,2015-11-19 09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user6_AliceBlueDing

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #95

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

[aaltay] [BEAM-3255] Enabling gradle-based release process (#5187)

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 863.22 KB...]
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.63.134:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.63.134:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches WritableServerSelector. Client view of cluster state is 
{type=UNKNOWN, servers=[{address=104.154.63.134:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.select

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #100

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-4147) Artifact source abstractions

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom updated BEAM-4147:
-
Summary: Artifact source abstractions  (was: Portable runner Job API 
abstractions)

> Artifact source abstractions
> 
>
> Key: BEAM-4147
> URL: https://issues.apache.org/jira/browse/BEAM-4147
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>
> We need a way to wire in arbitrary runner artifact storage backends into the 
> job server and through to artifact staging on workers. This requires some new 
> abstractions in front of the job service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #497

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

[aaltay] [BEAM-3255] Enabling gradle-based release process (#5187)

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 846.48 KB...]
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.23.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.23.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.23.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev233-1.23.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev124-1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.23.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.23.0 
from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ 
beam-sdks-java-io-jdbc ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.001 s 
<<< FAILURE! - in org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] org.apache.beam.sdk.io.jdbc.JdbcIOIT  Time elapsed: 0 s  <<< ERROR!
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory

Build failed in Jenkins: beam_PerformanceTests_Python #1193

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

[aaltay] [BEAM-3255] Enabling gradle-based release process (#5187)

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 816.84 KB...]
[INFO] Apache Beam :: Model ... SUCCESS [  0.029 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  5.413 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  2.890 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  3.534 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.145 s]
[INFO] Apache Beam :: SDKs :: Go .. SUCCESS [ 22.851 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SUCCESS [ 19.565 s]
[INFO] Apache Beam :: SDKs :: Java  SUCCESS [  0.133 s]
[INFO] Apache Beam :: SDKs :: Java :: Core  SUCCESS [ 21.859 s]
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SUCCESS [  2.874 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SUCCESS [  0.047 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SUCCESS [  3.290 s]
[INFO] Apache Beam :: Runners . SUCCESS [  0.061 s]
[INFO] Apache Beam :: Runners :: Core Construction Java ... SUCCESS [  4.354 s]
[INFO] Apache Beam :: Runners :: Core Java  SUCCESS [  4.227 s]
[INFO] Apache Beam :: SDKs :: Java :: Harness . SUCCESS [  8.126 s]
[INFO] Apache Beam :: SDKs :: Java :: Container ... SUCCESS [  7.438 s]
[INFO] Apache Beam :: SDKs :: Java :: IO .. SUCCESS [  0.060 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SUCCESS [  
2.670 s]
[INFO] Apache Beam :: Runners :: Local Java Core .. SUCCESS [  1.578 s]
[INFO] Apache Beam :: Runners :: Direct Java .. SUCCESS [ 10.542 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SUCCESS [  2.410 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Common  SUCCESS [  1.776 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Cassandra . SUCCESS [  2.721 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch . SUCCESS [  1.900 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests SUCCESS [  
0.525 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: Common 
SUCCESS [  0.589 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 2.x SUCCESS 
[  1.819 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: 5.x SUCCESS 
[  1.877 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: XML ... SUCCESS [  2.270 s]
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Protobuf SUCCESS [  2.497 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform SUCCESS [  
4.969 s]
[INFO] Apache Beam :: Runners :: Google Cloud Dataflow  SUCCESS [  8.706 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: File-based-io-tests SUCCESS [  
2.182 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Common . SUCCESS [  3.026 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop File System SUCCESS [  3.657 
s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [  3.051 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop Input Format SUCCESS [  
8.544 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HBase . SUCCESS [  5.564 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: HCatalog .. SUCCESS [  9.062 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: JMS ... SUCCESS [  2.212 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka . SUCCESS [  2.666 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis ... SUCCESS [  1.835 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB ... SUCCESS [  3.124 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: MQTT .. SUCCESS [  2.194 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Redis . SUCCESS [  2.542 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Solr .. SUCCESS [  4.551 s]
[INFO] Apache Beam :: SDKs :: Java :: IO :: Tika .. SUCCESS [  3.131 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [  0.021 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter SUCCESS [  
7.019 s]
[INFO] Apache Beam :: Examples ...

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #420

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

[aaltay] [BEAM-3255] Enabling gradle-based release process (#5187)

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 857.13 KB...]
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.23.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-bigquery:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-http:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-appengine:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.api:gax-httpjson:jar:0.37.0 from the shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.11.1 from the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-http-util:jar:0.11.1 from the 
shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev382-1.23.0 from the shaded 
jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the s

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #422

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

[aaltay] [BEAM-3255] Enabling gradle-based release process (#5187)

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 841.22 KB...]
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.23.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-bigquery:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-http:jar:1.23.0 from the 
shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-appengine:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.api:gax-httpjson:jar:0.37.0 from the shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.11.1 from the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-http-util:jar:0.11.1 from the 
shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev382-1.23.0 from the shaded 
jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.23.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.

[jira] [Commented] (BEAM-4130) Portable Flink runner entry point

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453265#comment-16453265
 ] 

Eugene Kirpichov commented on BEAM-4130:


Clarification: we want the entry point to be p.run(). E.g., p.run() should 
package the pipeline to a proto, start a local JobService, submit the proto to 
it, and the JobService should submit itself to the Flink master. Likewise in 
the Python and Go SDKs (the only language-specific code being packaging the 
pipeline to proto and starting the runner-specific JobService - the JobService 
itself being, of course, implemented once per runner).

> Portable Flink runner entry point
> -
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Minor
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4071) Portable Runner Job API shim

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-4071.
--
   Resolution: Fixed
Fix Version/s: 2.5.0

> Portable Runner Job API shim
> 
>
> Key: BEAM-4071
> URL: https://issues.apache.org/jira/browse/BEAM-4071
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 18h
>  Remaining Estimate: 0h
>
> There needs to be a way to execute Java-SDK pipelines against a portable job 
> server. The job server itself is expected to be started up out-of-band. The 
> "PortableRunner" should take an option indicating the Job API endpoint and 
> defer other runner configurations to the backend itself.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4176) Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom reassigned BEAM-4176:


Assignee: (was: Aljoscha Krettek)

> Portable batch runner passes all ValidatesRunner tests that non-portable 
> runner passes
> --
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4176) Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4176:


 Summary: Portable batch runner passes all ValidatesRunner tests 
that non-portable runner passes
 Key: BEAM-4176
 URL: https://issues.apache.org/jira/browse/BEAM-4176
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Aljoscha Krettek


We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4146) Python SDK sets environment in portable pipelines

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-4146.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> Python SDK sets environment in portable pipelines
> -
>
> Key: BEAM-4146
> URL: https://issues.apache.org/jira/browse/BEAM-4146
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> Environments must be set in any non-runner-executed transforms. See 
> [https://github.com/bsidhom/beam/commit/0362fd1f25] for a possible approach 
> until canonical image urls are created.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=95308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95308
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:56
Start Date: 25/Apr/18 23:56
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5228: [BEAM-4073] Expand 
ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228#issuecomment-384469505
 
 
   R: @huygaa11


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95308)
Time Spent: 20m  (was: 10m)

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4073) The DirectRunner should interact with a Pipeline via an abstraction of the Graph rather than SDK types

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4073?focusedWorklogId=95307&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95307
 ]

ASF GitHub Bot logged work on BEAM-4073:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:55
Start Date: 25/Apr/18 23:55
Worklog Time Spent: 10m 
  Work Description: tgroh opened a new pull request #5228: [BEAM-4073] 
Expand ExecutableGraph, and add a Portable Representation
URL: https://github.com/apache/beam/pull/5228
 
 
   This change adds additional methods required to execute a Pipeline to
   ExecutableGraph, a Portable implementation of the interface, and expands
   the implementation of `QueryablePipeline` to be able to fulfill that 
interface.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95307)
Time Spent: 10m
Remaining Estimate: 0h

> The DirectRunner should interact with a Pipeline via an abstraction of the 
> Graph rather than SDK types
> --
>
> Key: BEAM-4073
> URL: https://issues.apache.org/jira/browse/BEAM-4073
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3773) [SQL] Investigate JDBC interface for Beam SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=95306&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95306
 ]

ASF GitHub Bot logged work on BEAM-3773:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:49
Start Date: 25/Apr/18 23:49
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5173: 
[BEAM-3773][SQL] Add EnumerableConverter for JDBC support
URL: https://github.com/apache/beam/pull/5173#discussion_r184237912
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
 ##
 @@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.MetricNameFilter;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
+import org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.convert.ConverterImpl;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataType;
+
+/**
+ * BeamRelNode to replace a {@code Enumerable} node.
+ */
+public class BeamEnumerableConverter extends ConverterImpl implements 
EnumerableRel {
+
+  public BeamEnumerableConverter(RelOptCluster cluster, RelTraitSet traits, 
RelNode input) {
+super(cluster, ConventionTraitDef.INSTANCE, traits, input);
+  }
+
+  @Override
+  public RelNode copy(RelTraitSet traitSet, List inputs) {
+return new BeamEnumerableConverter(getCluster(), traitSet, sole(inputs));
+  }
+
+  @Override
+  public RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery 
mq) {
+// This should always be a last resort.
+return planner.getCostFactory().makeHugeCost();
+  }
+
+  @Override
+  public Result implement(EnumerableRelImplementor implementor, Prefer prefer) 
{
+final BlockBuilder list = new BlockBuilder();
+final RelDataType rowType = getRowType();
+final PhysType physType =
+PhysTypeImpl.of(implementor.getTypeFactory(), rowType, 
prefer.preferArray());
+final Expression node = implementor.stash((BeamRelNode) getInput(), 
BeamRelNode.class);
+list.add(Expressions.call(BeamEnumerableConverter.class, "toEnumerable", 
node));
+return implementor.result(physType, list.toBlock());
+  }
+
+  public static Enumerable toEnumerable(BeamRelNode node) {
+PipelineOptions options = PipelineOptionsFactory.create();
+if (node instanceof BeamIOSinkRel) {
+  return count(options, node);
+}
+return collect(options, node);
+  }
+
+  private static PipelineResult run(PipelineOptions options, BeamRelNode node,
+

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #190

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Moves standard URNs into .proto files

[ekirpichov] Refactors common_urns.py

[ekirpichov] Fixes pylint errors

[ekirpichov] Fixes compile errors

[ekirpichov] Workaround for protobuf compiler bug

[ekirpichov] Fixes more references to common_urns

--
[...truncated 19.18 MB...]
org.apache.beam.examples.cookbook.FilterExamplesTest > testProjectionFn 
STANDARD_ERROR
Apr 25, 2018 11:41:07 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractCountryInfoFn 
STANDARD_ERROR
Apr 25, 2018 11:41:07 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.cookbook.JoinExamplesTest > testExtractEventDataFn 
STANDARD_ERROR
Apr 25, 2018 11:41:07 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.DebuggingWordCountTest > testDebuggingWordCount 
STANDARD_ERROR
Apr 25, 2018 11:41:08 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
/tmp/junit3542244491470037721/junit6723968775814615438.tmp matched 1 files with 
total size 54
Apr 25, 2018 11:41:08 PM org.apache.beam.sdk.io.FileBasedSource split
INFO: Splitting filepattern 
/tmp/junit3542244491470037721/junit6723968775814615438.tmp into bundles of size 
3 took 1 ms and produced 1 files and 18 bundles

org.apache.beam.examples.WordCountTest > testExtractWordsFn STANDARD_ERROR
Apr 25, 2018 11:41:08 PM org.apache.beam.sdk.transforms.DoFnTester of
WARNING: Your tests use DoFnTester, which may not exercise DoFns correctly. 
Please use TestPipeline instead.

org.apache.beam.examples.subprocess.ExampleEchoPipelineTest > 
testExampleEchoPipeline STANDARD_ERROR
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-Echo3575475036455062336.sh 
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-Echo3575475036455062336.sh to 
/tmp/test-Echoo360632392480774011/test-Echo3575475036455062336.sh 
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils initSemaphore
INFO: Initialized Semaphore for binary test-EchoAgain648595034353412101.sh 
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.CallingSubProcessUtils setUp
INFO: Calling filesetup to move Executables to worker.
Apr 25, 2018 11:41:10 PM 
org.apache.beam.examples.subprocess.utils.FileUtils copyFileFromGCSToWorker
INFO: Moving File /tmp/test-EchoAgain648595034353412101.sh to 
/tmp/test-Echoo360632392480774011/test-EchoAgain648595034353412101.sh 

org.apache.beam.examples.complete.game.HourlyTeamScoreTest > 
testUserScoresFilter STANDARD_OUT
GOT user18_ApricotCaneToad,ApricotCaneToad,14,144796569,2015-11-19 
12:41:31.053
GOT user18_BananaEmu,BananaEmu,1,144796569,2015-11-19 12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,8,144795563,2015-11-19 09:53:53.444
GOT user18_BananaEmu,BananaEmu,7,144796569,2015-11-19 12:41:31.053
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,11,144795563,2015-11-19 
09:53:53.444
GOT user7_AlmondWallaby,AlmondWallaby,15,144795563,2015-11-19 
09:53:53.444
GOT user13_ApricotQuokka,ApricotQuokka,15,144795563,2015-11-19 
09:53:53.444
GOT user2_AmberCockatoo,AmberCockatoo,13,144796569,2015-11-19 
12:41:31.053
GOT user19_BisqueBilby,BisqueBilby,6,144795563,2015-11-19 09:53:53.444
GOT user6_AmberNumbat,AmberNumbat,11,144795563,2015-11-19 09:53:53.444
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT user3_BananaEmu,BananaEmu,17,144796569,2015-11-19 12:41:31.053
GOT user0_MagentaKangaroo,MagentaKangaroo,4,144796569,2015-11-19 
12:41:31.053
GOT 
user0_AndroidGreenEchidna,AndroidGreenEchidna,0,144796569,2015-11-19 
12:41:31.053

org.apache.beam.examples.complete.game.UserScoreTest > testTeamScoreSums 
STANDARD_OUT
GOT user0_MagentaKangaroo,MagentaKangaroo,3,144795563,2015-11-19 
09:53:53.444
GOT 
user7_AndroidGreenKookaburra,AndroidGreenKookaburra,12,144795563,2015-1

[jira] [Assigned] (BEAM-3947) Add support for Solr 6.x/7.x

[Cao Manh Dat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cao Manh Dat reassigned BEAM-3947:
--

Assignee: Cao Manh Dat

> Add support for Solr 6.x/7.x
> 
>
> Key: BEAM-3947
> URL: https://issues.apache.org/jira/browse/BEAM-3947
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-solr
>Reporter: Ismaël Mejía
>Assignee: Cao Manh Dat
>Priority: Minor
>
> The initial PR on Solr was based on Solr 6.x, however at that time we 
> supported Java 7 so Insisted to move it to Solr 5.x (which is Java 7 
> compatible). This issue is to add support for multiple versions of Solr 
> ideally in a single module.
> Notice that I was able to recover the original code for Solr 6.x created by 
> [~caomanhdat] here (there are some differences in the way the Split was 
> calculated and maybe some other minor things):(
> [https://github.com/iemejia/beam/blob/recover-solrio/sdks/java/io/solr/pom.xml]
> This issue does not cover support for Solr 7, but if it is possible to add it 
> as part of it, it would be great.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3773) [SQL] Investigate JDBC interface for Beam SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=95300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95300
 ]

ASF GitHub Bot logged work on BEAM-3773:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:30
Start Date: 25/Apr/18 23:30
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #5173: 
[BEAM-3773][SQL] Add EnumerableConverter for JDBC support
URL: https://github.com/apache/beam/pull/5173#discussion_r184234873
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
 ##
 @@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.MetricNameFilter;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
+import org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.calcite.linq4j.Enumerable;
+import org.apache.calcite.linq4j.Linq4j;
+import org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.calcite.linq4j.tree.Expression;
+import org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.convert.ConverterImpl;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataType;
+
+/**
+ * BeamRelNode to replace a {@code Enumerable} node.
+ */
+public class BeamEnumerableConverter extends ConverterImpl implements 
EnumerableRel {
+
+  public BeamEnumerableConverter(RelOptCluster cluster, RelTraitSet traits, 
RelNode input) {
+super(cluster, ConventionTraitDef.INSTANCE, traits, input);
+  }
+
+  @Override
+  public RelNode copy(RelTraitSet traitSet, List inputs) {
+return new BeamEnumerableConverter(getCluster(), traitSet, sole(inputs));
+  }
+
+  @Override
+  public RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery 
mq) {
+// This should always be a last resort.
+return planner.getCostFactory().makeHugeCost();
+  }
+
+  @Override
+  public Result implement(EnumerableRelImplementor implementor, Prefer prefer) 
{
+final BlockBuilder list = new BlockBuilder();
+final RelDataType rowType = getRowType();
+final PhysType physType =
+PhysTypeImpl.of(implementor.getTypeFactory(), rowType, 
prefer.preferArray());
+final Expression node = implementor.stash((BeamRelNode) getInput(), 
BeamRelNode.class);
+list.add(Expressions.call(BeamEnumerableConverter.class, "toEnumerable", 
node));
+return implementor.result(physType, list.toBlock());
+  }
+
+  public static Enumerable toEnumerable(BeamRelNode node) {
+PipelineOptions options = PipelineOptionsFactory.create();
 
 Review comment:
   Plumbing the `PipelineOptions` into `BeamEnumerableConverter` is not easy as 
they will have to take a trip through the calcite JDBC interface on the way 
here (possibly on the jdbc connection string). I can add them as an

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=95297&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95297
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:12
Start Date: 25/Apr/18 23:12
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5224: [BEAM-4044] [SQL] 
Cleanout unneeded sqlEnv
URL: https://github.com/apache/beam/pull/5224#issuecomment-384462294
 
 
   R: @XuMingmin @xumingming This is a trivial change, but it does change the 
public interface around BeamQueryPlanner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95297)
Time Spent: 8h 50m  (was: 8h 40m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4672: Moves standard URNs into .proto files

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit bbabb4202cdbe234ab7be9a1cda57f0e8e226357
Merge: 5bb482f 58502e0
Author: Eugene Kirpichov 
AuthorDate: Wed Apr 25 16:08:06 2018 -0700

Merge pull request #4672: Moves standard URNs into .proto files

Moves standard URNs into .proto files

 .gitignore |   1 -
 .../beam/model/fnexecution/v1/standard_coders.yaml |  56 ++---
 .../pipeline/src/main/proto/beam_runner_api.proto  | 141 
 .../src/main/proto/standard_window_fns.proto   |  25 ++-
 .../resources/org/apache/beam/model/common_urns.md | 134 
 pom.xml|   1 -
 runners/core-construction-java/build.gradle|   2 +
 runners/core-construction-java/pom.xml |  13 ++
 .../beam/runners/core/construction/BeamUrns.java   |  29 +++
 .../core/construction/ModelCoderRegistrar.java |   4 +
 .../runners/core/construction/ModelCoders.java |  19 +-
 .../core/construction/PTransformTranslation.java   |  41 ++--
 .../beam/runners/core/construction/UrnUtils.java   |  63 --
 .../construction/WindowingStrategyTranslation.java |  82 +++
 .../core/construction}/CommonCoderTest.java| 241 +++--
 .../runners/core/construction/UrnUtilsTest.java|  59 -
 runners/java-fn-execution/build.gradle |   1 +
 .../control/ProcessBundleDescriptors.java  |   5 +-
 .../graph/LengthPrefixUnknownCoders.java   |  18 +-
 sdks/java/core/build.gradle|   1 +
 .../beam/fn/harness/WindowMappingFnRunner.java |   6 +-
 sdks/python/apache_beam/coders/coders.py   |  18 +-
 .../apache_beam/coders/standard_coders_test.py |  28 +--
 sdks/python/apache_beam/io/iobase.py   |   4 +-
 sdks/python/apache_beam/pipeline.py|   2 +-
 sdks/python/apache_beam/portability/common_urns.py |  62 ++
 sdks/python/apache_beam/pvalue.py  |  10 +-
 .../runners/dataflow/dataflow_runner.py|  12 +-
 .../runners/portability/fn_api_runner.py   |  50 ++---
 .../apache_beam/runners/worker/bundle_processor.py |  24 +-
 .../apache_beam/testing/data/standard_coders.yaml  |  56 ++---
 sdks/python/apache_beam/transforms/core.py |  25 ++-
 sdks/python/apache_beam/transforms/window.py   |  16 +-
 sdks/python/setup.py   |  31 ---
 34 files changed, 632 insertions(+), 648 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (5bb482f -> bbabb42)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5bb482f  Merge pull request #5150:  [BEAM-4071] Add Portable Runner 
Job API shim
 add 7adcb3c  Moves standard URNs into .proto files
 add d1d2b92  Refactors common_urns.py
 add eba19c12 Fixes pylint errors
 add ac94f25  Fixes compile errors
 add 42fac77  Workaround for protobuf compiler bug
 add 58502e0  Fixes more references to common_urns
 new bbabb42  Merge pull request #4672: Moves standard URNs into .proto 
files

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .gitignore |   1 -
 .../beam/model/fnexecution/v1/standard_coders.yaml |  56 ++---
 .../pipeline/src/main/proto/beam_runner_api.proto  | 141 
 .../src/main/proto/standard_window_fns.proto   |  25 ++-
 .../resources/org/apache/beam/model/common_urns.md | 134 
 pom.xml|   1 -
 runners/core-construction-java/build.gradle|   2 +
 runners/core-construction-java/pom.xml |  13 ++
 .../{package-info.java => BeamUrns.java}   |  17 +-
 .../core/construction/ModelCoderRegistrar.java |   4 +
 .../runners/core/construction/ModelCoders.java |  19 +-
 .../core/construction/PTransformTranslation.java   |  41 ++--
 .../beam/runners/core/construction/UrnUtils.java   |  63 --
 .../construction/WindowingStrategyTranslation.java |  82 +++
 .../core/construction}/CommonCoderTest.java| 241 +++--
 .../runners/core/construction/UrnUtilsTest.java|  59 -
 runners/java-fn-execution/build.gradle |   1 +
 .../control/ProcessBundleDescriptors.java  |   5 +-
 .../graph/LengthPrefixUnknownCoders.java   |  18 +-
 sdks/java/core/build.gradle|   1 +
 .../beam/fn/harness/WindowMappingFnRunner.java |   6 +-
 sdks/python/apache_beam/coders/coders.py   |  18 +-
 .../apache_beam/coders/standard_coders_test.py |  28 +--
 sdks/python/apache_beam/io/iobase.py   |   4 +-
 sdks/python/apache_beam/pipeline.py|   2 +-
 sdks/python/apache_beam/portability/common_urns.py |  62 ++
 sdks/python/apache_beam/pvalue.py  |  10 +-
 .../runners/dataflow/dataflow_runner.py|  12 +-
 .../runners/portability/fn_api_runner.py   |  50 ++---
 .../apache_beam/runners/worker/bundle_processor.py |  24 +-
 .../apache_beam/testing/data/standard_coders.yaml  |  56 ++---
 sdks/python/apache_beam/transforms/core.py |  25 ++-
 sdks/python/apache_beam/transforms/window.py   |  16 +-
 sdks/python/setup.py   |  31 ---
 34 files changed, 613 insertions(+), 655 deletions(-)
 delete mode 100644 
model/pipeline/src/main/resources/org/apache/beam/model/common_urns.md
 copy 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{package-info.java
 => BeamUrns.java} (66%)
 delete mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UrnUtils.java
 rename {sdks/java/core/src/test/java/org/apache/beam/sdk/coders => 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction}/CommonCoderTest.java
 (59%)
 delete mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UrnUtilsTest.java
 create mode 100644 sdks/python/apache_beam/portability/common_urns.py

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=95294&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95294
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:02
Start Date: 25/Apr/18 23:02
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r184229759
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StageBundleFactory.java
 ##
 @@ -0,0 +1,16 @@
+package org.apache.beam.runners.fnexecution.control;
+
+/**
+ * A bundle factory scoped to a particular
+ * {@link org.apache.beam.runners.core.construction.graph.ExecutableStage}, 
which has all of the
+ * resources it needs to provide new {@link RemoteBundle RemoteBundles}.
+ *
+ * Closing a StageBundleFactory signals that a a stage has completed and 
any resources bound to
+ * its lifetime can be cleaned up.
+ */
+public interface StageBundleFactory extends AutoCloseable {
+  /**
+   * Get a new {@link RemoteBundle bundle} for processing the data in an 
executable stage.
+   */
+   RemoteBundle getBundle() throws Exception;
 
 Review comment:
   How do clients register bundle output receivers? Note that when interacting 
with the `SdkHarnessClient` directly, this is done via `newBundle`:
   
   
https://github.com/apache/beam/blob/5bb482f536e75bfacac91a546ab3db843dd2a25e/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java#L114
   
   Note that we cannot register output receivers at operator startup because 
the output receivers are attached their respective partition contexts (i.e., in 
the case of Flink outputs are sunk to the `Collector` which can only be used 
for the lifetime of a partition (`mapPartition` function call)).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95294)
Time Spent: 22h 40m  (was: 22.5h)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>  Time Spent: 22h 40m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=95293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95293
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:02
Start Date: 25/Apr/18 23:02
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r184227583
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/JobBundleFactory.java
 ##
 @@ -0,0 +1,13 @@
+package org.apache.beam.runners.fnexecution.control;
 
 Review comment:
   Please add a license header. Same applies to the other new files below.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95293)
Time Spent: 22.5h  (was: 22h 20m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>  Time Spent: 22.5h
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=95295&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95295
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:02
Start Date: 25/Apr/18 23:02
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r184228593
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
 ##
 @@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+import io.grpc.stub.StreamObserver;
+import java.io.IOException;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
+
+/**
+ * Makes artifacts available to an ArtifactRetrievalService by
+ * encapsulating runner-specific resources.
+ */
+public interface ArtifactSource {
+
+  /**
+   * Get the artifact manifest available from this source.
+   */
+  Manifest getManifest() throws IOException;
+
+  /**
+   * Get an artifact by its name.
+   */
+  void getArtifact(String name, StreamObserver 
responseObserver);
 
 Review comment:
   Do we want to make clients interact directly with StreamObserver? This may 
be the most reasonable approach, but it also increases the burden on 
implementors due to gRPC semantics. For example, clients may need to wrap code 
to ensure that `onCompleted` or `onError` is always called to make sure RPCs 
complete in a timely fashion. If this is necessary, please add a doc comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95295)
Time Spent: 22h 50m  (was: 22h 40m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>  Time Spent: 22h 50m
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3327) Add abstractions to manage Environment Instance lifecycles.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=95296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95296
 ]

ASF GitHub Bot logged work on BEAM-3327:


Author: ASF GitHub Bot
Created on: 25/Apr/18 23:02
Start Date: 25/Apr/18 23:02
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5152: 
[BEAM-3327] Harness Manager Interfaces
URL: https://github.com/apache/beam/pull/5152#discussion_r184230277
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/StaticBundleFactory.java
 ##
 @@ -0,0 +1,14 @@
+package org.apache.beam.runners.fnexecution.control;
+
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+
+/**
+ * A factory that can create {@link JobBundleFactory JobBundleFactories} from 
job-scoped resources
+ * provided by an operator.
+ *
+ * This should be a global or singleton resource that caches 
JobBundleFactories between calls.
 
 Review comment:
   Is the `StaticBundleFactory` itself meant to be a global singleton or is the 
retuned result meant to be singleton scoped (per job)? This might be clearer as 
a class rather than interface.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95296)
Time Spent: 23h  (was: 22h 50m)

> Add abstractions to manage Environment Instance lifecycles.
> ---
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
>  Time Spent: 23h
>  Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #189

[Apache Jenkins Server]

See 


Changes:

[sidhom] Add in-process ManagedChannelFactory and update tests to use it

[sidhom] Add common pipeline options for portable runners

[sidhom] Set artifact names explicitly while staging to service

[sidhom] Add CloseableResource to wrap non-closeable resources that must be

[sidhom] [BEAM-4071] Add Portable Runner Job API shim

--
[...truncated 20.11 MB...]
Apr 25, 2018 10:55:15 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-common/4.1.8.Final/netty-common-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/netty-common-4.1.8.Final-lafOW8vGsI6SRwZSmXNClg.jar
Apr 25, 2018 10:55:15 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/com/google/auto/value/auto-value/1.4/auto-value-1.4.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/auto-value-1.4-_7tYdqNvCowycFIDpBenFw.jar
Apr 25, 2018 10:55:15 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 

 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/beam-sdks-java-io-google-cloud-platform-2.5.0-SNAPSHOT-shaded-tests-0H_bINqh474db9AuqdxujA.jar
Apr 25, 2018 10:55:15 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/io/netty/netty-handler-proxy/4.1.8.Final/netty-handler-proxy-4.1.8.Final.jar
 to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/netty-handler-proxy-4.1.8.Final-Zey48Fj4mlWtgpwIc7Osgg.jar
Apr 25, 2018 10:55:15 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/home/jenkins/.m2/repository/com/squareup/okhttp/okhttp/2.5.0/okhttp-2.5.0.jar 
to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/okhttp-2.5.0-64v0X4G_nxfR_PsuymOqpg.jar
Apr 25, 2018 10:55:20 PM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Staging files complete: 0 files cached, 127 files newly uploaded
Apr 25, 2018 10:55:20 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding GenerateSequence/Read(BoundedCountingSource) as step s1
Apr 25, 2018 10:55:20 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParDo(CreateEntity) as step s2
Apr 25, 2018 10:55:20 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding DatastoreV1.Write/Convert to Mutation/Map as step s3
Apr 25, 2018 10:55:20 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding DatastoreV1.Write/Write Mutation to Datastore as step s4
Apr 25, 2018 10:55:20 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/
Apr 25, 2018 10:55:20 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <8774 bytes, hash c-CZlcnyOzIczDEKFLuWYQ> to 
gs://temp-storage-for-end-to-end-tests/testpipeline-jenkins-0425225513-56bd4e2e/output/results/staging/pipeline-c-CZlcnyOzIczDEKFLuWYQ.pb

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_ERROR
Apr 25, 2018 10:55:22 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-25_15_55_21-10465330941772867377?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_OUT
Submitted job: 2018-04-25_15_55_21-10465330941772867377

org.apache.beam.sdk.io.gcp.datastore.V1WriteIT > testE2EV1Write STANDARD_ERROR
Apr 25, 2018 10:55:22 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-04-25_15_55_21-10465330941772867377
Apr 25, 2018 10:55:22 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-04-2

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=95284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95284
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 25/Apr/18 22:20
Start Date: 25/Apr/18 22:20
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5135: [BEAM-4028] 
Transitioning MapTask objects to NameContext
URL: https://github.com/apache/beam/pull/5135#issuecomment-384452304
 
 
   I believe this is ready to merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95284)
Time Spent: 5h  (was: 4h 50m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=95281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95281
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 25/Apr/18 22:18
Start Date: 25/Apr/18 22:18
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-384451792
 
 
   After collecting additional datapoints on the same benchmark, the regression 
seems to be on the order of 7%, I am doing more A/B testing to confirm the 
exact changes that contribute to it.  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95281)
Time Spent: 14h 50m  (was: 14h 40m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 14h 50m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5150: [BEAM-4071] Add Portable Runner Job API shim

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5bb482f536e75bfacac91a546ab3db843dd2a25e
Merge: b195516 5d5c052
Author: Eugene Kirpichov 
AuthorDate: Wed Apr 25 15:09:38 2018 -0700

Merge pull request #5150:  [BEAM-4071] Add Portable Runner Job API shim

 [BEAM-4071] Add Portable Runner Job API shim

 .../core/construction/ArtifactServiceStager.java   |  58 +++--
 .../construction/ArtifactServiceStagerTest.java|   9 +-
 .../runners/fnexecution/InProcessSdkHarness.java   |  11 +-
 .../fnexecution/control/RemoteExecutionTest.java   |  12 +-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |  10 +-
 runners/reference/java/build.gradle|   5 +-
 .../beam/runners/reference/CloseableResource.java  | 120 ++
 .../reference/JobServicePipelineResult.java| 147 +
 .../beam/runners/reference/PortableRunner.java | 245 +
 .../testing/InProcessManagedChannelFactory.java}   |  28 +--
 .../runners/reference/testing/TestJobService.java  |  78 +++
 .../runners/reference/testing/package-info.java}   |  18 +-
 .../runners/reference/CloseableResourceTest.java   | 109 +
 .../beam/runners/reference/PortableRunnerTest.java |  84 +++
 .../job/ReferenceRunnerJobServiceTest.java |  10 +-
 .../beam/sdk/options/PortablePipelineOptions.java  |  46 
 .../fn/test/InProcessManagedChannelFactory.java|  28 +--
 17 files changed, 928 insertions(+), 90 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-4071) Portable Runner Job API shim

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4071?focusedWorklogId=95277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95277
 ]

ASF GitHub Bot logged work on BEAM-4071:


Author: ASF GitHub Bot
Created on: 25/Apr/18 22:09
Start Date: 25/Apr/18 22:09
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #5150:  [BEAM-4071] Add 
Portable Runner Job API shim
URL: https://github.com/apache/beam/pull/5150
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
index 7319b8f30d1..63cc50810c3 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
@@ -47,6 +47,7 @@
 import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
 import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
 import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
+import 
org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
 import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
 import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
 import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
@@ -87,26 +88,33 @@ private ArtifactServiceStager(Channel channel, int 
bufferSize) {
 this.bufferSize = bufferSize;
   }
 
-  public void stage(Iterable files) throws IOException, 
InterruptedException {
-final Map> futures = new 
HashMap<>();
-for (File file : files) {
+  /**
+   * Stages the given artifact files to the staging service.
+   *
+   * @return The artifact staging token returned by the service
+   */
+  public String stage(Iterable files) throws IOException, 
InterruptedException {
+final Map> futures = new 
HashMap<>();
+for (StagedFile file : files) {
   futures.put(file, MoreFutures.supplyAsync(new StagingCallable(file), 
executorService));
 }
 CompletionStage stagingResult =
 MoreFutures.allAsList(futures.values())
 .thenApply(ignored -> new 
ExtractStagingResultsCallable(futures).call());
-stageManifest(stagingResult);
+return stageManifest(stagingResult);
   }
 
-  private void stageManifest(CompletionStage stagingFuture)
+  private String stageManifest(CompletionStage stagingFuture)
   throws InterruptedException {
 try {
   StagingResult stagingResult = MoreFutures.get(stagingFuture);
   if (stagingResult.isSuccess()) {
 Manifest manifest =
 
Manifest.newBuilder().addAllArtifact(stagingResult.getMetadata()).build();
-blockingStub.commitManifest(
-CommitManifestRequest.newBuilder().setManifest(manifest).build());
+CommitManifestResponse response =
+blockingStub.commitManifest(
+
CommitManifestRequest.newBuilder().setManifest(manifest).build());
+return response.getStagingToken();
   } else {
 RuntimeException failure =
 new RuntimeException(
@@ -124,9 +132,9 @@ private void stageManifest(CompletionStage 
stagingFuture)
   }
 
   private class StagingCallable implements ThrowingSupplier {
-private final File file;
+private final StagedFile file;
 
-private StagingCallable(File file) {
+private StagingCallable(StagedFile file) {
   this.file = file;
 }
 
@@ -135,11 +143,12 @@ public ArtifactMetadata get() throws Exception {
   // TODO: Add Retries
   PutArtifactResponseObserver responseObserver = new 
PutArtifactResponseObserver();
   StreamObserver requestObserver = 
stub.putArtifact(responseObserver);
-  ArtifactMetadata metadata = 
ArtifactMetadata.newBuilder().setName(file.getName()).build();
+  ArtifactMetadata metadata =
+  ArtifactMetadata.newBuilder().setName(file.getStagingName()).build();
   
requestObserver.onNext(PutArtifactRequest.newBuilder().setMetadata(metadata).build());
 
   MessageDigest md5Digest = MessageDigest.getInstance("MD5");
-  FileChannel channel = new FileInputStream(file).getChannel();
+  FileChannel channel = new FileInputStream(file.getFile()).getChannel();
   ByteBuffer readBuffer = ByteBuffer.allocate(bufferSize);
   while (!responseObserver.isTerminal() && channel.position() < 
channel.size()) {
 readBuffer.clea

[beam] branch master updated (b195516 -> 5bb482f)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b195516  [BEAM-3255] Enabling gradle-based release process (#5187)
 add f7a7018  Add in-process ManagedChannelFactory and update tests to use 
it
 add e5d0c8b  Add common pipeline options for portable runners
 add fa7b0c0  Set artifact names explicitly while staging to service
 add 87bdc1c  Add CloseableResource to wrap non-closeable resources that 
must be cleaned up
 add 5d5c052  [BEAM-4071] Add Portable Runner Job API shim
 new 5bb482f  Merge pull request #5150:  [BEAM-4071] Add Portable Runner 
Job API shim

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/ArtifactServiceStager.java   |  58 +++--
 .../construction/ArtifactServiceStagerTest.java|   9 +-
 .../runners/fnexecution/InProcessSdkHarness.java   |  11 +-
 .../fnexecution/control/RemoteExecutionTest.java   |  12 +-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |  10 +-
 runners/reference/java/build.gradle|   5 +-
 .../beam/runners/reference/CloseableResource.java  | 120 ++
 .../reference/JobServicePipelineResult.java| 147 +
 .../beam/runners/reference/PortableRunner.java | 245 +
 .../testing/InProcessManagedChannelFactory.java|  24 +-
 .../runners/reference/testing/TestJobService.java  |  78 +++
 .../runners/reference/testing}/package-info.java   |   6 +-
 .../runners/reference/CloseableResourceTest.java   | 109 +
 .../beam/runners/reference/PortableRunnerTest.java |  84 +++
 .../job/ReferenceRunnerJobServiceTest.java |  10 +-
 .../beam/sdk/options/PortablePipelineOptions.java  |  46 
 ...fo.java => InProcessManagedChannelFactory.java} |  20 +-
 17 files changed, 926 insertions(+), 68 deletions(-)
 create mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
 create mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
 create mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
 copy 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
 => 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InProcessManagedChannelFactory.java
 (54%)
 create mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
 copy 
runners/{gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils
 => 
reference/java/src/main/java/org/apache/beam/runners/reference/testing}/package-info.java
 (88%)
 create mode 100644 
runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
 create mode 100644 
runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
 copy 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/{package-info.java
 => InProcessManagedChannelFactory.java} (56%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam-site] branch asf-site updated (9772653 -> 3acc115)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 9772653  This closes #429
 add 09f47f4  Friendlier contribution guide; link to starter tasks
 add da98d3c  This closes #426
 new 3acc115  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/contribution-guide/index.html | 34 +++-
 src/contribute/contribution-guide.md | 33 +--
 2 files changed, 52 insertions(+), 15 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 3acc1156ef6cb937115d12470927ff518fceed8d
Author: Mergebot 
AuthorDate: Wed Apr 25 14:59:39 2018 -0700

Prepare repository for deployment.
---
 content/contribute/contribution-guide/index.html | 34 +++-
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/content/contribute/contribution-guide/index.html 
b/content/contribute/contribution-guide/index.html
index ff7b90e..55f2d9f 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -145,6 +145,7 @@
 
 
 
+  Review-then-commit 
workflow
   Engage
 
   Mailing list(s)
@@ -179,14 +180,33 @@
   
 Apache Beam Contribution 
Guide
 
-The Apache Beam community welcomes contributions from anyone with a passion 
for
-data processing! Beam has many different opportunities for contributions –
-write new examples, add new user-facing libraries (new statistical libraries,
-new IO connectors, etc), work on the core programming model, build specific
-runners (Apache Apex, Apache Flink, Apache Spark, Google Cloud Dataflow, etc),
-or participate on the documentation effort.
+The Apache Beam community welcomes contributions from anyone!
 
-We use a review-then-commit workflow in Beam for all contributions.
+There are lots of opportunities:
+
+
+  write new examples
+  improve the documentation
+  add new user-facing libraries (new statistical libraries, new IO 
connectors,
+etc)
+  improve your favorite language SDK (Java, Python, Go, etc)
+  improve specific runners (Apache Apex, Apache Flink, Apache Spark, Google
+Cloud Dataflow, etc)
+  work on the core programming model (what is a Beam pipeline and how does 
it
+run?)
+
+
+Most importantly, if you have an idea of how to contribute, then do it!
+
+For a list of open starter tasks, check
+https://s.apache.org/beam-starter-tasks";>https://s.apache.org/beam-starter-tasks.
+
+What follows is a more comprehensive document covering details of things 
that
+may come up.
+
+Review-then-commit workflow
+
+We use a review-then-commit workflow in Beam.
 
 
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #426

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit da98d3c959db67cfa1ab420bf59a91f26985a504
Merge: 9772653 09f47f4
Author: Mergebot 
AuthorDate: Wed Apr 25 14:54:47 2018 -0700

This closes #426

 src/contribute/contribution-guide.md | 33 +
 1 file changed, 25 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Friendlier contribution guide; link to starter tasks

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 09f47f4c67cd8313378f1fd7957928e170dcbba1
Author: Kenneth Knowles 
AuthorDate: Sat Apr 21 11:50:45 2018 -0700

Friendlier contribution guide; link to starter tasks
---
 src/contribute/contribution-guide.md | 33 +
 1 file changed, 25 insertions(+), 8 deletions(-)

diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
index 68cb082..44ea14b 100644
--- a/src/contribute/contribution-guide.md
+++ b/src/contribute/contribution-guide.md
@@ -8,14 +8,31 @@ redirect_from: /contribution-guide/
 
 # Apache Beam Contribution Guide
 
-The Apache Beam community welcomes contributions from anyone with a passion for
-data processing! Beam has many different opportunities for contributions --
-write new examples, add new user-facing libraries (new statistical libraries,
-new IO connectors, etc), work on the core programming model, build specific
-runners (Apache Apex, Apache Flink, Apache Spark, Google Cloud Dataflow, etc),
-or participate on the documentation effort.
-
-We use a review-then-commit workflow in Beam for all contributions.
+The Apache Beam community welcomes contributions from anyone!
+
+There are lots of opportunities:
+
+ - write new examples
+ - improve the documentation
+ - add new user-facing libraries (new statistical libraries, new IO connectors,
+   etc)
+ - improve your favorite language SDK (Java, Python, Go, etc)
+ - improve specific runners (Apache Apex, Apache Flink, Apache Spark, Google
+   Cloud Dataflow, etc)
+ - work on the core programming model (what is a Beam pipeline and how does it
+   run?)
+
+Most importantly, if you have an idea of how to contribute, then do it! 
+
+For a list of open starter tasks, check
+[https://s.apache.org/beam-starter-tasks](https://s.apache.org/beam-starter-tasks).
+
+What follows is a more comprehensive document covering details of things that
+may come up.
+
+## Review-then-commit workflow
+
+We use a review-then-commit workflow in Beam.
 
 ![The Beam contribution workflow has 5 steps: engage, design, code, review, 
and commit.](
   {{ "/images/contribution-guide-1.png" | prepend: site.baseurl }})

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-3485) CassandraIO.read() splitting produces invalid queries

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3485?focusedWorklogId=95275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95275
 ]

ASF GitHub Bot logged work on BEAM-3485:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:55
Start Date: 25/Apr/18 21:55
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5124: 
[BEAM-3485] Fix split generation for Cassandra clusters
URL: https://github.com/apache/beam/pull/5124#discussion_r184217860
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/SplitGenerator.java
 ##
 @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.google.common.collect.Lists;
+import java.math.BigInteger;
+import java.util.List;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Splits given Cassandra table's token range into splits.
+ */
+final class SplitGenerator {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(SplitGenerator.class);
+
+  private final String partitioner;
+  private final BigInteger rangeMin;
+  private final BigInteger rangeMax;
+  private final BigInteger rangeSize;
+
+  SplitGenerator(String partitioner){
+rangeMin = getRangeMin(partitioner);
+rangeMax = getRangeMax(partitioner);
+rangeSize = getRangeSize(partitioner);
+this.partitioner = partitioner;
+  }
+
+  SplitGenerator(BigInteger rangeMin, BigInteger rangeMax) {
+this.rangeMin = rangeMin;
+this.rangeMax = rangeMax;
+rangeSize = rangeMax.subtract(rangeMin).add(BigInteger.ONE);
+partitioner = "(" + rangeMin + "," + rangeMax + ")";
+  }
+
+  static BigInteger getRangeMin(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return BigInteger.ZERO;
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).negate();
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeMax(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return new BigInteger("2").pow(127).subtract(BigInteger.ONE);
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).subtract(BigInteger.ONE);
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeSize(String partitioner) {
+return 
getRangeMax(partitioner).subtract(getRangeMin(partitioner)).add(BigInteger.ONE);
+  }
+
+  static BigInteger max(BigInteger big0, BigInteger big1) {
+return greaterThan(big0, big1) ? big0 : big1;
+  }
+
+  static BigInteger min(BigInteger big0, BigInteger big1) {
+return lowerThan(big0, big1) ? big0 : big1;
+  }
+
+  static boolean lowerThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) < 0;
+  }
+
+  static boolean lowerThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) <= 0;
+  }
+
+  static boolean greaterThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) > 0;
+  }
+
+  static boolean greaterThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) >= 0;
+  }
+
+  /**
+   * Given big0 properly ordered list of tokens, compute at least {@code 
totalSplitCount} splits.
+   *
+   * @param totalSplitCount requested total amount of splits. This function 
may generate
+   *more splits.
+   * @param ringTokens list of all start tokens in big0 cluster. They have to 
be in ring order.
+   * @return big0 list containing at least {@code totalSplitCount} splits.
+   */
+  List generateSplits(long totalSplitCount, List 
ringTokens) {
+
+int tokenRangeCount = ringTokens.size();
 
 Review comment:
   @jkff since you are quite good to find tricky corner cases would you mind to 
take a look just at this method (I will take ca

[beam-site] branch mergebot updated (6cdec24 -> da98d3c)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 6cdec24  This closes #429
 add 9772653  This closes #429
 new 09f47f4  Friendlier contribution guide; link to starter tasks
 new da98d3c  This closes #426

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (6cdec24)
\
 N -- N -- N   refs/heads/mergebot (da98d3c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/contribute/contribution-guide.md | 33 +
 1 file changed, 25 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=95274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95274
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:53
Start Date: 25/Apr/18 21:53
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5135: [BEAM-4028] 
Transitioning MapTask objects to NameContext
URL: https://github.com/apache/beam/pull/5135#issuecomment-384446462
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95274)
Time Spent: 4h 50m  (was: 4h 40m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3746) Count.globally should override getIncompatibleGlobalWindowErrorMessage to tell the user the usage that is currently only in javadoc

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3746?focusedWorklogId=95272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95272
 ]

ASF GitHub Bot logged work on BEAM-3746:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:52
Start Date: 25/Apr/18 21:52
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5176: [BEAM-3746] Added 
error message for using Count.globally() on non-global windows
URL: https://github.com/apache/beam/pull/5176#issuecomment-384446089
 
 
   I see - there is no override, because Count does not encapsulate itself. 
That's a bit of a mess.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95272)
Time Spent: 50m  (was: 40m)

> Count.globally should override getIncompatibleGlobalWindowErrorMessage to 
> tell the user the usage that is currently only in javadoc
> ---
>
> Key: BEAM-3746
> URL: https://issues.apache.org/jira/browse/BEAM-3746
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>  Labels: beginner, newbie, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/transforms/Count.html#globally--
> "Note: if the input collection uses a windowing strategy other than 
> GlobalWindows, use Combine.globally(Count.combineFn()).withoutDefaults() 
> instead."
> But the actual crash a user gets is:
> "java.lang.IllegalStateException: Default values are not supported in 
> Combine.globally() if the output PCollection is not windowed by 
> GlobalWindows. Instead, use Combine.globally().withoutDefaults() to output an 
> empty PCollection if the input PCollection is empty, or 
> Combine.globally().asSingletonView() to get the default output of the 
> CombineFn if the input PCollection is empty."
> There is a method that exists solely to make this actually useful, so we 
> should use it!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3485) CassandraIO.read() splitting produces invalid queries

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3485?focusedWorklogId=95271&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95271
 ]

ASF GitHub Bot logged work on BEAM-3485:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:51
Start Date: 25/Apr/18 21:51
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5124: 
[BEAM-3485] Fix split generation for Cassandra clusters
URL: https://github.com/apache/beam/pull/5124#discussion_r184216720
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/SplitGenerator.java
 ##
 @@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.cassandra;
+
+import com.google.common.collect.Lists;
+import java.math.BigInteger;
+import java.util.List;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Splits given Cassandra table's token range into splits.
+ */
+final class SplitGenerator {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(SplitGenerator.class);
+
+  private final String partitioner;
+  private final BigInteger rangeMin;
+  private final BigInteger rangeMax;
+  private final BigInteger rangeSize;
+
+  SplitGenerator(String partitioner){
+rangeMin = getRangeMin(partitioner);
+rangeMax = getRangeMax(partitioner);
+rangeSize = getRangeSize(partitioner);
+this.partitioner = partitioner;
+  }
+
+  SplitGenerator(BigInteger rangeMin, BigInteger rangeMax) {
+this.rangeMin = rangeMin;
+this.rangeMax = rangeMax;
+rangeSize = rangeMax.subtract(rangeMin).add(BigInteger.ONE);
+partitioner = "(" + rangeMin + "," + rangeMax + ")";
+  }
+
+  static BigInteger getRangeMin(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return BigInteger.ZERO;
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).negate();
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeMax(String partitioner) {
+if (partitioner.endsWith("RandomPartitioner")) {
+  return new BigInteger("2").pow(127).subtract(BigInteger.ONE);
+} else if (partitioner.endsWith("Murmur3Partitioner")) {
+  return new BigInteger("2").pow(63).subtract(BigInteger.ONE);
+} else {
+  throw new UnsupportedOperationException("Unsupported partitioner. "
+  + "Only Random and Murmur3 are supported");
+}
+  }
+
+  static BigInteger getRangeSize(String partitioner) {
+return 
getRangeMax(partitioner).subtract(getRangeMin(partitioner)).add(BigInteger.ONE);
+  }
+
+  static BigInteger max(BigInteger big0, BigInteger big1) {
+return greaterThan(big0, big1) ? big0 : big1;
+  }
+
+  static BigInteger min(BigInteger big0, BigInteger big1) {
+return lowerThan(big0, big1) ? big0 : big1;
+  }
+
+  static boolean lowerThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) < 0;
+  }
+
+  static boolean lowerThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) <= 0;
+  }
+
+  static boolean greaterThan(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) > 0;
+  }
+
+  static boolean greaterThanOrEqual(BigInteger big0, BigInteger big1) {
+return big0.compareTo(big1) >= 0;
+  }
+
+  /**
+   * Given big0 properly ordered list of tokens, compute at least {@code 
totalSplitCount} splits.
+   *
+   * @param totalSplitCount requested total amount of splits. This function 
may generate
+   *more splits.
+   * @param ringTokens list of all start tokens in big0 cluster. They have to 
be in ring order.
+   * @return big0 list containing at least {@code totalSplitCount} splits.
+   */
+  List generateSplits(long totalSplitCount, List 
ringTokens) {
+
+int tokenRangeCount = ringTokens.size();
 
 Review comment:
   I found an inconsistency in the split production to reproduce it run the 
CassandraIOIT tests (you can hardcore the options t

[jira] [Work logged] (BEAM-3303) Go windowing support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3303?focusedWorklogId=95267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95267
 ]

ASF GitHub Bot logged work on BEAM-3303:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:50
Start Date: 25/Apr/18 21:50
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5179: 
[BEAM-3303] Add Go Windowing support
URL: https://github.com/apache/beam/pull/5179#discussion_r184214639
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/translate.go
 ##
 @@ -454,50 +457,52 @@ func stepID(id int) string {
return fmt.Sprintf("s%v", id)
 }
 
-func translateWindow(w *window.Window) proto.Message {
-   // TODO: The only windowing strategy we support is the global window.
-   if w.Kind() != window.GlobalWindow {
-   panic(fmt.Sprintf("Unsupported window type supplied: %v", w))
-   }
-   // We compute the fixed content of this message for use in workflows.
-   msg := rnapi_pb.MessageWithComponents{
+func translateWindowingStrategy(w *window.WindowingStrategy) proto.Message {
+   c := graphx.NewCoderMarshaller()
+   ws := graphx.MarshalWindowingStrategy(c, w)
+
+   msg := &rnapi_pb.MessageWithComponents{
Components: &rnapi_pb.Components{
-   Coders: map[string]*rnapi_pb.Coder{
-   "Coder": &rnapi_pb.Coder{
-   Spec: &rnapi_pb.SdkFunctionSpec{
-   Spec: &rnapi_pb.FunctionSpec{
-   Urn: 
"urn:beam:coders:global_window:0.1",
-   },
-   },
-   },
-   },
+   Coders: c.Build(),
},
Root: &rnapi_pb.MessageWithComponents_WindowingStrategy{
-   WindowingStrategy: &rnapi_pb.WindowingStrategy{
-   WindowFn: &rnapi_pb.SdkFunctionSpec{
-   Spec: &rnapi_pb.FunctionSpec{
-   Urn: 
"beam:windowfn:global_windows:v0.1",
-   },
-   },
-   MergeStatus:  
rnapi_pb.MergeStatus_NON_MERGING,
-   AccumulationMode: 
rnapi_pb.AccumulationMode_DISCARDING,
-   WindowCoderId:"Coder",
-   Trigger: &rnapi_pb.Trigger{
-   Trigger: &rnapi_pb.Trigger_Default_{
-   Default: 
&rnapi_pb.Trigger_Default{},
-   },
-   },
-   OutputTime:  
rnapi_pb.OutputTime_END_OF_WINDOW,
-   ClosingBehavior: 
rnapi_pb.ClosingBehavior_EMIT_IF_NONEMPTY,
-   AllowedLateness: 0,
-   },
+   WindowingStrategy: ws,
},
}
-
-   return &msg
+   return msg
 }
 
 func encodeSerializedFn(in proto.Message) (string, error) {
-   // The Beam Runner API uses URL query escaping for serialized fn 
messages.
-   return protox.EncodeQueryEscaped(in)
+   // The Beam Runner API uses special escaping for serialized fn messages.
+
+   data, err := proto.Marshal(in)
+   if err != nil {
+   return "", err
+   }
+   return encodeString(data), nil
+}
+
+// encodeString is a custom encoding used in some cases by Dataflow.
+//
+// Uses a simple strategy of converting each byte to a single char,
 
 Review comment:
   This is defined as 'percent encoding' or less accurately as 'URL encoding.' 
   https://en.wikipedia.org/wiki/Percent-encoding
   
   Pretty sure this is identical to url.PathEscape, perhaps use that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95267)
Time Spent: 3h 50m  (was: 3h 40m)

> Go windowing support
> 
>
> Key: BEAM-3303
> URL: https://issues.apache.org/jira/browse/BEAM-3303
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h

[jira] [Work logged] (BEAM-3303) Go windowing support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3303?focusedWorklogId=95263&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95263
 ]

ASF GitHub Bot logged work on BEAM-3303:


Author: ASF GitHub Bot
Created on: 25/Apr/18 21:50
Start Date: 25/Apr/18 21:50
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #5179: 
[BEAM-3303] Add Go Windowing support
URL: https://github.com/apache/beam/pull/5179#discussion_r184211558
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/graphx/coder.go
 ##
 @@ -75,15 +74,15 @@ type CoderUnmarshaller struct {
models map[string]*pb.Coder
 
coders  map[string]*coder.Coder
-   windows map[string]*window.Window
+   windows map[string]*coder.WindowCoder
 
 Review comment:
   windowCoders?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 95263)
Time Spent: 3.5h  (was: 3h 20m)

> Go windowing support
> 
>
> Key: BEAM-3303
> URL: https://issues.apache.org/jira/browse/BEAM-3303
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Add support for Window.into and windowing strategies on Node. Implement the 
> various windowing strategies Beam has: GlobalWindow, FixedWindows, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)