[jira] [Commented] (BEAM-3456) Enable large scale JdbcIOIT Performance Test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330176#comment-16330176
 ] 

ASF GitHub Bot commented on BEAM-3456:
--

chamikaramj closed pull request #4392: [BEAM-3456] Enable jenkins and large 
scale scenario in JDBC
URL: https://github.com/apache/beam/pull/4392
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index e7b475d4caa..b86020ec278 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -91,10 +91,10 @@
 
   /* Options for test pipeline for file-based I/O in 
'sdks/java/io/file-based-io-tests/'. */
   @Description("Number records that will be written and read by the test")
-  @Default.Long(10)
-  Long getNumberOfRecords();
+  @Default.Integer(10)
+  Integer getNumberOfRecords();
 
-  void setNumberOfRecords(Long count);
+  void setNumberOfRecords(Integer count);
 
   @Description("Destination prefix for files generated by the test")
   @Validation.Required
diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
index 5f0a2fb00b2..79a144d144d 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
@@ -95,7 +95,9 @@ public void processElement(ProcessContext c) {
* the name() for the rows generated from seeds in [0, n).
*/
   private static final Map EXPECTED_HASHES = ImmutableMap.of(
-  1000, "7d94d63a41164be058a9680002914358"
+  1000, "7d94d63a41164be058a9680002914358",
+  100_000, "c7cbddb319209e200f1c5eebef8fe960",
+  5_000_000, "c44f8a5648cd9207c9c6f77395a998dc"
   );
 
   /**
diff --git 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
index be0d6df2eb7..07562f38ca3 100644
--- 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
+++ 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
@@ -75,7 +75,7 @@
   + "}");
 
   private static String filenamePrefix;
-  private static Long numberOfTextLines;
+  private static Integer numberOfTextLines;
 
   @Rule
   public TestPipeline pipeline = TestPipeline.create();
diff --git 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
index cf20d8e5954..40b04617d8a 100644
--- 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
+++ 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
@@ -55,11 +55,11 @@ public static String appendTimestampToPrefix(String 
filenamePrefix) {
 return String.format("%s_%s", filenamePrefix, new Date().getTime());
   }
 
-  public static String getExpectedHashForLineCount(Long lineCount) {
-Map expectedHashes = ImmutableMap.of(
-100_000L, "4c8bb3b99dcc59459b20fefba400d446",
-1_000_000L, "9796db06e7a7960f974d5a91164afff1",
-100_000_000L, "6ce05f456e2fdc846ded2abd0ec1de95"
+  public static String getExpectedHashForLineCount(int lineCount) {
+Map expectedHashes = ImmutableMap.of(
+100_000, "4c8bb3b99dcc59459b20fefba400d446",
+1_000_000, "9796db06e7a7960f974d5a91164afff1",
+100_000_000, "6ce05f456e2fdc846ded2abd0ec1de95"
 );
 
 String hash = expectedHashes.get(lineCount);
diff --git 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
index 1a4e0ef..b611a5746d3 100644
--- 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
+++ 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
@@ -23,7 +23,6 @@
 import static 
org.apache.beam.sdk.io.common.FileBasedIOITHelper.getExpectedHashForLineCount;
 import static 
org.apache.beam.sdk.io.common.FileBasedIOITHelper.readTestPipelineOptions;
 
-import 

[beam] branch master updated: [BEAM-3456] Enable jenkins and large scale scenario in JDBC (#4392)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2f235dd  [BEAM-3456] Enable jenkins and large scale scenario in JDBC 
(#4392)
2f235dd is described below

commit 2f235dd58acce27f713c7072d62cd3a72e3413a1
Author: Łukasz Gajowy 
AuthorDate: Thu Jan 18 08:20:48 2018 +0100

[BEAM-3456] Enable jenkins and large scale scenario in JDBC (#4392)

[BEAM-3456] Enable jenkins and large scale scenario in JDBC

The kubernetes infrastructure that is needed for the
Jenkins job to run is not available for now.
We should add it once the infrastructure is there.
---
 .../beam/sdk/io/common/IOTestPipelineOptions.java  |  6 +++---
 .../org/apache/beam/sdk/io/common/TestRow.java |  4 +++-
 .../java/org/apache/beam/sdk/io/avro/AvroIOIT.java |  2 +-
 .../beam/sdk/io/common/FileBasedIOITHelper.java| 10 +-
 .../java/org/apache/beam/sdk/io/text/TextIOIT.java |  5 ++---
 .../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java  |  5 ++---
 sdks/java/io/jdbc/pom.xml  |  2 ++
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 23 ++
 8 files changed, 28 insertions(+), 29 deletions(-)

diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index e7b475d..b86020e 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -91,10 +91,10 @@ public interface IOTestPipelineOptions extends 
TestPipelineOptions {
 
   /* Options for test pipeline for file-based I/O in 
'sdks/java/io/file-based-io-tests/'. */
   @Description("Number records that will be written and read by the test")
-  @Default.Long(10)
-  Long getNumberOfRecords();
+  @Default.Integer(10)
+  Integer getNumberOfRecords();
 
-  void setNumberOfRecords(Long count);
+  void setNumberOfRecords(Integer count);
 
   @Description("Destination prefix for files generated by the test")
   @Validation.Required
diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
index 5f0a2fb..79a144d 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
@@ -95,7 +95,9 @@ public abstract class TestRow implements Serializable, 
Comparable {
* the name() for the rows generated from seeds in [0, n).
*/
   private static final Map EXPECTED_HASHES = ImmutableMap.of(
-  1000, "7d94d63a41164be058a9680002914358"
+  1000, "7d94d63a41164be058a9680002914358",
+  100_000, "c7cbddb319209e200f1c5eebef8fe960",
+  5_000_000, "c44f8a5648cd9207c9c6f77395a998dc"
   );
 
   /**
diff --git 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
index be0d6df..07562f3 100644
--- 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
+++ 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
@@ -75,7 +75,7 @@ public class AvroIOIT {
   + "}");
 
   private static String filenamePrefix;
-  private static Long numberOfTextLines;
+  private static Integer numberOfTextLines;
 
   @Rule
   public TestPipeline pipeline = TestPipeline.create();
diff --git 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
index cf20d8e..40b0461 100644
--- 
a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
+++ 
b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
@@ -55,11 +55,11 @@ public class FileBasedIOITHelper {
 return String.format("%s_%s", filenamePrefix, new Date().getTime());
   }
 
-  public static String getExpectedHashForLineCount(Long lineCount) {
-Map expectedHashes = ImmutableMap.of(
-100_000L, "4c8bb3b99dcc59459b20fefba400d446",
-1_000_000L, "9796db06e7a7960f974d5a91164afff1",
-100_000_000L, "6ce05f456e2fdc846ded2abd0ec1de95"
+  public static String getExpectedHashForLineCount(int lineCount) {
+Map expectedHashes = ImmutableMap.of(
+100_000, "4c8bb3b99dcc59459b20fefba400d446",
+1_000_000, 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #5680

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3487) GroupByKey stalls with GroupingShuffleReader split refusals

[Bei Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330099#comment-16330099
 ] 

Bei Zhang commented on BEAM-3487:
-

Also at one point it throws an error "Refusing to split  at 
ShufflePosition(base64:AmGGbvcAAQ): proposed split position out of range". 
This happens when using {code}WriteFiles.to{code}.

> GroupByKey stalls with GroupingShuffleReader split refusals
> ---
>
> Key: BEAM-3487
> URL: https://issues.apache.org/jira/browse/BEAM-3487
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Bei Zhang
>Assignee: Thomas Groh
>Priority: Major
>
> With info messages with something like:
> {quote}{{Refused to split GroupingShuffleReader  [ShufflePosition(base64:A1CWNvgAAQ), 
> ShufflePosition(base64:A4sOz1AAAQ))> at 
> ShufflePosition(base64:A1CWNvkAAQ)}}
> {quote}
> The lull messages look like this:
> {quote}{{Processing lull for PT300.006S in state read-shuffle of Write 
> Vectors2/GroupIntoShards/Read at 
> com.google.cloud.dataflow.worker.ApplianceShuffleReader.readIncludingPosition(Native
>  Method) at 
> com.google.cloud.dataflow.worker.ChunkingShuffleBatchReader.read(ChunkingShuffleBatchReader.java:62)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:57)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:53)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>  at 
> com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader.read(CachingShuffleBatchReader.java:76)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:133)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntriesIfNeeded(BatchingShuffleEntryReader.java:126)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.hasNext(BatchingShuffleEntryReader.java:90)
>  at 
> com.google.cloud.dataflow.worker.util.common.ForwardingReiterator.hasNext(ForwardingReiterator.java:62)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.GroupingShuffleEntryIterator.advance(GroupingShuffleEntryIterator.java:118)
>  at 
> com.google.cloud.dataflow.worker.GroupingShuffleReader$GroupingShuffleReaderIterator.advance(GroupingShuffleReader.java:230)
>  at 
> com.google.cloud.dataflow.worker.GroupingShuffleReader$GroupingShuffleReaderIterator.start(GroupingShuffleReader.java:224)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:347)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:183)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:148)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:68)
>  at 
> com.google.cloud.dataflow.worker.DataflowWorker.executeWork(DataflowWorker.java:330)
>  at 
> com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:302)
>  at 
> com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:251)
>  at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
>  at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
>  at 
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> 

Build failed in Jenkins: beam_PerformanceTests_Python #804

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated 
PR

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b53100b335eefcb27ec9d0c124fafdda9fda11a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53100b335eefcb27ec9d0c124fafdda9fda11a4
Commit message: "Merge pull request #4432 from jkff/rm-max-bundle-size"
 > git rev-list 40c9081d4152462630558b12faa39d62a640d030 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1054261046170647193.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8754832115759458899.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2497388286970322645.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4970884326582752251.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8312985358667767324.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #28

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #29

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated 
PR

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b53100b335eefcb27ec9d0c124fafdda9fda11a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53100b335eefcb27ec9d0c124fafdda9fda11a4
Commit message: "Merge pull request #4432 from jkff/rm-max-bundle-size"
 > git rev-list 40c9081d4152462630558b12faa39d62a640d030 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins805845230495158423.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6139277553674077780.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4245687657692866567.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3044991650404615023.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins415600503123987.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6752134977019864536.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #36

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated 
PR

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b53100b335eefcb27ec9d0c124fafdda9fda11a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53100b335eefcb27ec9d0c124fafdda9fda11a4
Commit message: "Merge pull request #4432 from jkff/rm-max-bundle-size"
 > git rev-list 40c9081d4152462630558b12faa39d62a640d030 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins9095628062660296118.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6473764258444842230.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4679327347990748677.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins898436318603851318.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8878421202303405284.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1809055365925178241.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 

Build failed in Jenkins: beam_PerformanceTests_Spark #1248

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated 
PR

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b53100b335eefcb27ec9d0c124fafdda9fda11a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53100b335eefcb27ec9d0c124fafdda9fda11a4
Commit message: "Merge pull request #4432 from jkff/rm-max-bundle-size"
 > git rev-list 40c9081d4152462630558b12faa39d62a640d030 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7374538943439962358.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4982226036299867736.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5390054739283821126.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3685143901873013332.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2811068108697889443.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #28

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] Removes maxBundleSize option accidentally introduced in unrelated 
PR

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision b53100b335eefcb27ec9d0c124fafdda9fda11a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b53100b335eefcb27ec9d0c124fafdda9fda11a4
Commit message: "Merge pull request #4432 from jkff/rm-max-bundle-size"
 > git rev-list 40c9081d4152462630558b12faa39d62a640d030 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins543867338148933.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5268544083919104665.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3902818802233637220.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3968114099606702437.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7645308397078104982.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3041507571657822518.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 

[jira] [Commented] (BEAM-3494) Snapshot state of aggregated data of apache beam project is not maintained in flink's checkpointing

[suganya (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330077#comment-16330077
 ] 

suganya commented on BEAM-3494:
---

he following code is using apache beam libraries to create a pipeline.Please 
find the code.

public void run(String[] args)

{ BeamCLIOptions beamCliOptions = 
PipelineOptionsFactory.fromArgs(args).withValidation() 
.as(BeamCLIOptions.class); Pipeline pipeline = Pipeline.create(beamCliOptions); 
MergeDistribution mergeDistribution = MergeDistribution 
.valueOf(beamCliOptions.getMergeDistribution()); MergeDistribution 
fixedWindowDuration = MergeDistribution 
.valueOf(beamCliOptions.getFixedWindowSize()); KafkaIO.Read 
kafkaEntityStreamReader = KafkaIO.read() 
.withBootstrapServers(beamCliOptions.getKafkaServers()) 
.withTopic(beamCliOptions.getKafkaTopic()) 
.withKeyDeserializer(StringDeserializer.class) 
.withValueDeserializer(StringDeserializer.class) 
.updateConsumerProperties(ImmutableMap.of("auto.offset.reset", 
"latest","enable.auto.commit","true")); 
pipeline.apply(kafkaEntityStreamReader.withoutMetadata()) 
.apply(Values.create()) .apply(Window.into( 
FixedWindows.of(Duration.standardMinutes(fixedWindowDuration.getMins( 
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane() 
.plusDelayOf(Duration.standardMinutes(mergeDistribution.getMins() 
.discardingFiredPanes() .withAllowedLateness(Duration.ZERO)) 
.apply(ParDo.of(new ExtractDataFn( beamCliOptions.getDatePartitionKey(), new 
DateTime().minusDays(beamCliOptions.getDaysAgo()).getMillis( 
.apply("Applying GroupByKey on -MM-DD HH ", GroupByKey.create()) 
.apply("Applying Merge ", ParDo.of(new MergeDataFn(beamCliOptions))); 
pipeline.run(); }

> Snapshot state of aggregated data of apache beam project is not maintained in 
> flink's checkpointing 
> 
>
> Key: BEAM-3494
> URL: https://issues.apache.org/jira/browse/BEAM-3494
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: suganya
>Assignee: Kenneth Knowles
>Priority: Major
>
> We have a beam project which consumes events from kafka,does a groupby in a 
> time window(5 mins),after window elapses it pushes the events to downstream 
> for merge.This project is deployed using flink ,we have enabled checkpointing 
> to recover from failed state.
> (windowsize: 5mins , checkpointingInterval: 5mins,state.backend: filesystem)
> Offsets from kafka get checkpointed every 5 
> mins(checkpointingInterval).Before finishing the entire DAG(groupBy and 
> merge) , events offsets are getting checkpointed.So incase of any restart 
> from task-manager ,new task gets started from last successful checkpoint ,but 
> we could'nt able to get the aggregated snapshot data(data from groupBy task) 
> from the persisted checkpoint.
> Able to retrieve the last successful checkpointed offset from kafka ,but 
> couldnt able to get last aggregated data till checkpointing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3494) Snapshot state of aggregated data of apache beam project is not maintained in flink's checkpointing

[suganya (JIRA)]

suganya created BEAM-3494:
-

 Summary: Snapshot state of aggregated data of apache beam project 
is not maintained in flink's checkpointing 
 Key: BEAM-3494
 URL: https://issues.apache.org/jira/browse/BEAM-3494
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: suganya
Assignee: Kenneth Knowles


We have a beam project which consumes events from kafka,does a groupby in a 
time window(5 mins),after window elapses it pushes the events to downstream for 
merge.This project is deployed using flink ,we have enabled checkpointing to 
recover from failed state.

(windowsize: 5mins , checkpointingInterval: 5mins,state.backend: filesystem)

Offsets from kafka get checkpointed every 5 mins(checkpointingInterval).Before 
finishing the entire DAG(groupBy and merge) , events offsets are getting 
checkpointed.So incase of any restart from task-manager ,new task gets started 
from last successful checkpoint ,but we could'nt able to get the aggregated 
snapshot data(data from groupBy task) from the persisted checkpoint.

Able to retrieve the last successful checkpointed offset from kafka ,but 
couldnt able to get last aggregated data till checkpointing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #4739

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329998#comment-16329998
 ] 

Kenneth Knowles commented on BEAM-3414:
---

Can I give it to you Aljoscha, since you have that PR just about in?

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Aljoscha Krettek
>Priority: Major
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3414:
-

Assignee: Aljoscha Krettek  (was: Kenneth Knowles)

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Aljoscha Krettek
>Priority: Major
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3493) Prevent users from "implementing" PipelineOptions

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3493:
-

 Summary: Prevent users from "implementing" PipelineOptions
 Key: BEAM-3493
 URL: https://issues.apache.org/jira/browse/BEAM-3493
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Luke Cwik


I've seen a user implement \{{PipelineOptions}}. This implies that it is 
backwards-incompatible to add new options, which is of course not our intent. 
We should at least document very loudly that it is not to be implemented, and 
preferably have some automation that will fail on load if they have implemented 
it. Ideas?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5679

[Apache Jenkins Server]

See 

[jira] [Issue Comment Deleted] (BEAM-3414) AfterProcessingTime trigger issue with Flink Runner

[huangjianhuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huangjianhuang updated BEAM-3414:
-
Comment: was deleted

(was: Thanks for help. Does this bug appear in other Runner? Or can you give me 
some advice which Runner is more close to the DirectRunner. My codes works fine 
with DirectRunner but got so many problems with FlinkRunner:()

> AfterProcessingTime trigger issue with Flink Runner
> ---
>
> Key: BEAM-3414
> URL: https://issues.apache.org/jira/browse/BEAM-3414
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.2.0
> Environment: idea, ubuntu 16.04, FlinkRunner
>Reporter: huangjianhuang
>Assignee: Kenneth Knowles
>Priority: Major
>
> in my demo, I read data from kafka and count globally, finally output the 
> total count of recieved data, as follow:
> {code:java}
> FlinkPipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation()
> .as(FlinkPipelineOptions.class);
> options.setStreaming(true);
> options.setRunner(FlinkRunner.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
> .apply("Read from kafka",
> KafkaIO.read()
> //.withTimestampFn(kafkaData -> 
> TimeUtil.timeMillisToInstant(kafkaData.getKey()))
> .withBootstrapServers("localhost:9092")
> .withTopic("recharge")
> .withKeyDeserializer(StringDeserializer.class)
> 
> .withValueDeserializer(StringDeserializer.class)
> .withoutMetadata()
> )
> .apply(Values.create())
> .apply(Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> 
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(5
> .accumulatingFiredPanes()
> )
> .apply(Count.globally())
> .apply("output",
> ParDo.of(new DoFn() {
> @ProcessElement
> public void process(ProcessContext context) {
> System.out.println("---get at: " + 
> Instant.now() + "--");
> System.out.println(context.element());
> }
> }));
> {code}
> the result should be displayed after (5s) I sent first data, but sometimes 
> there were nothing display after I sent data. the pic shows the outputs i got 
> in a test:
> (cant upload a pic, desc as text)
> {code:java}
> Send 681Msg at: 2018-01-05T06:34:31.436
>   ---get at: 2018-01-05T06:34:36.668Z--
>   681
> Send 681Msg at: 2018-01-05T06:34:47.166
>   ---get at: 2018-01-05T06:34:52.284Z--
>   1362
> Send 681Msg at: 2018-01-05T06:34:55.505
> Send 681Msg at: 2018-01-05T06:35:22.068
>   ---get at: 2018-01-05T06:35:22.112Z--
>   2044
> {code}
> btw, the code works fine with direct runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4738

[Apache Jenkins Server]

See 


--
[...truncated 22.86 MB...]
[INFO] 2018-01-18T02:02:04.267Z: (1c7eb2542bc3e3ed): Unzipping flatten s15 for 
input s10.10
[INFO] 2018-01-18T02:02:04.283Z: (1c7eb2542bc3eebb): Fusing unzipped copy of 
PAssert$367/GroupGlobally/GroupDummyAndContents/Reify, through flatten 
PAssert$367/GroupGlobally/FlattenDummyAndContents, into producer 
PAssert$367/GroupGlobally/KeyForDummy/AddKeys/Map
[INFO] 2018-01-18T02:02:04.307Z: (1c7eb2542bc3e989): Unzipping flatten s15-u22 
for input s17-reify-value0-c20
[INFO] 2018-01-18T02:02:04.330Z: (1c7eb2542bc3e457): Fusing unzipped copy of 
PAssert$367/GroupGlobally/GroupDummyAndContents/Write, through flatten s15-u22, 
into producer PAssert$367/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2018-01-18T02:02:04.358Z: (1c7eb2542bc3ef25): Fusing consumer 
PAssert$367/GetPane/Map into PAssert$367/GroupGlobally/ParDo(Concat)
[INFO] 2018-01-18T02:02:04.392Z: (1c7eb2542bc3e9f3): Fusing consumer 
PAssert$367/VerifyAssertions/ParDo(DefaultConclude) into PAssert$367/RunChecks
[INFO] 2018-01-18T02:02:04.413Z: (1c7eb2542bc3e4c1): Fusing consumer 
PAssert$367/RunChecks into PAssert$367/GetPane/Map
[INFO] 2018-01-18T02:02:04.435Z: (1c7eb2542bc3ef8f): Fusing consumer 
PAssert$367/GroupGlobally/GroupDummyAndContents/GroupByWindow into 
PAssert$367/GroupGlobally/GroupDummyAndContents/Read
[INFO] 2018-01-18T02:02:04.455Z: (1c7eb2542bc3ea5d): Fusing consumer 
PAssert$367/GroupGlobally/Values/Values/Map into 
PAssert$367/GroupGlobally/GroupDummyAndContents/GroupByWindow
[INFO] 2018-01-18T02:02:04.477Z: (1c7eb2542bc3e52b): Fusing consumer 
PAssert$367/GroupGlobally/ParDo(Concat) into 
PAssert$367/GroupGlobally/Values/Values/Map
[INFO] 2018-01-18T02:02:04.504Z: (1c7eb2542bc3eff9): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) into 
PAssert$367/GroupGlobally/Window.Into()/Window.Assign
[INFO] 2018-01-18T02:02:04.525Z: (1c7eb2542bc3eac7): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/Write into 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/Reify
[INFO] 2018-01-18T02:02:04.548Z: (1c7eb2542bc3e595): Fusing consumer 
PAssert$367/GroupGlobally/RewindowActuals/Window.Assign into 
PAssert$367/GroupGlobally/GatherAllOutputs/Values/Values/Map
[INFO] 2018-01-18T02:02:04.570Z: (1c7eb2542bc3e063): Fusing consumer 
KvSwap/KvSwap/Map into Create.Values/Read(CreateSource)
[INFO] 2018-01-18T02:02:04.584Z: (1c7eb2542bc3eb31): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/Reify into 
PAssert$367/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign
[INFO] 2018-01-18T02:02:04.601Z: (1c7eb2542bc3e5ff): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map into 
PAssert$367/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)
[INFO] 2018-01-18T02:02:04.627Z: (1c7eb2542bc3e0cd): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow into 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/Read
[INFO] 2018-01-18T02:02:04.645Z: (1c7eb2542bc3eb9b): Fusing consumer 
PAssert$367/GroupGlobally/KeyForDummy/AddKeys/Map into 
PAssert$367/GroupGlobally/RewindowActuals/Window.Assign
[INFO] 2018-01-18T02:02:04.671Z: (1c7eb2542bc3e669): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign into 
PAssert$367/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map
[INFO] 2018-01-18T02:02:04.687Z: (1c7eb2542bc3e137): Fusing consumer 
PAssert$367/GroupGlobally/GatherAllOutputs/Values/Values/Map into 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/GroupByWindow
[INFO] 2018-01-18T02:02:04.713Z: (1c7eb2542bc3ec05): Fusing consumer 
PAssert$367/GroupGlobally/Window.Into()/Window.Assign into KvSwap/KvSwap/Map
[INFO] 2018-01-18T02:02:04.739Z: (1c7eb2542bc3e6d3): Fusing consumer 
PAssert$367/GroupGlobally/GroupDummyAndContents/Reify into 
PAssert$367/GroupGlobally/WindowIntoDummy/Window.Assign
[INFO] 2018-01-18T02:02:04.765Z: (1c7eb2542bc3e1a1): Fusing consumer 
PAssert$367/GroupGlobally/GroupDummyAndContents/Write into 
PAssert$367/GroupGlobally/GroupDummyAndContents/Reify
[INFO] 2018-01-18T02:02:04.782Z: (1c7eb2542bc3ec6f): Fusing consumer 
PAssert$367/GroupGlobally/WindowIntoDummy/Window.Assign into 
PAssert$367/GroupGlobally/Create.Values/Read(CreateSource)
[INFO] 2018-01-18T02:02:05.065Z: (efe6da217e0c): Executing operation 
PAssert$367/GroupGlobally/GatherAllOutputs/GroupByKey/Create
[INFO] 2018-01-18T02:02:05.119Z: (dd8a58028b2889de): Starting 1 workers in 
us-central1-f...
[INFO] 2018-01-18T02:02:05.245Z: (7ccbdf4f47d513b6): Executing operation 

[beam] branch master updated (40c9081 -> b53100b)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 40c9081  Merge pull request #4386
 add b764d36  Removes maxBundleSize option accidentally introduced in 
unrelated PR #4185
 new b53100b  Merge pull request #4432 from jkff/rm-max-bundle-size

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/java/org/apache/beam/sdk/options/PipelineOptions.java| 4 
 1 file changed, 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4432 from jkff/rm-max-bundle-size

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b53100b335eefcb27ec9d0c124fafdda9fda11a4
Merge: 40c9081 b764d36
Author: Eugene Kirpichov 
AuthorDate: Wed Jan 17 17:40:41 2018 -0800

Merge pull request #4432 from jkff/rm-max-bundle-size

Removes maxBundleSize option accidentally introduced in unrelated PR #4185

 .../src/main/java/org/apache/beam/sdk/options/PipelineOptions.java| 4 
 1 file changed, 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-2715) Expose PubsubSource to create UnboundedSource and utilize withMaxNumRecords from BoundedReadFromUnboundedSource

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329848#comment-16329848
 ] 

Thomas Groh commented on BEAM-2715:
---

Effectively, yes. Reading from a PubSub subscription either:
 * Produces duplicates, as no message is ACKed
 * Permanently removes the elements from the subscription

As a result, pulling from a real subscription has ill-defined behavior.

The actual change to support this method is relatively minor (but continues to 
hide the Pubsub Source). Example diff (admittedly untested) at the following 
link:

https://github.com/apache/beam/compare/master...tgroh:pubsub_max_num_records?expand=1

> Expose PubsubSource to create UnboundedSource and utilize withMaxNumRecords 
> from BoundedReadFromUnboundedSource
> ---
>
> Key: BEAM-2715
> URL: https://issues.apache.org/jira/browse/BEAM-2715
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-direct
>Reporter: Adam Levy
>Assignee: Thomas Groh
>Priority: Major
>  Labels: pubsub
>
> We are ingesting from a Pubsub Read and are attempting to mimic the 
> maxNumRecords that was available in 0.6.0. In order to do this we would need 
> to utilize withMaxNumRecords from the BoundedReadFromUnboundedSource class. 
> We would need to utilize the PubsubSource class to create the UnboundedSource 
> from Pubsub. Would it be possible to expose PubsubSource? Currently what is 
> the recommended way to create a bounded read from Pubsub with a set number of 
> records?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3450) RemoteGrpcPorts should contain the wire format

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3450.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> RemoteGrpcPorts should contain the wire format
> --
>
> Key: BEAM-3450
> URL: https://issues.apache.org/jira/browse/BEAM-3450
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: 2.3.0
>
>
> This forces the runner to include the wire format independently from a 
> PCollection coder, which should be the coder of the type of the PCollection 
> (e.g. in Java, PCollection has Coder instead of 
> Coder, but the runner must use Coder over 
> edges).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3492) Spark Integration Tests fail with a Closed Connection

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3492:
-

 Summary: Spark Integration Tests fail with a Closed Connection
 Key: BEAM-3492
 URL: https://issues.apache.org/jira/browse/BEAM-3492
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Thomas Groh
Assignee: Amit Sela


Example: [https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/16832]

 
2018-01-17T23:52:25.668 [ERROR] 
testE2EWordCount(org.apache.beam.examples.WordCountIT)  Time elapsed: 14.329 s  
<<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: 
Connection from /127.0.0.1:45363 closed
at 
org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:68)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3491) Issue with package of file and location

[Ron Gonzalez (JIRA)]

Ron Gonzalez created BEAM-3491:
--

 Summary: Issue with package of file and location
 Key: BEAM-3491
 URL: https://issues.apache.org/jira/browse/BEAM-3491
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Affects Versions: 2.2.0
Reporter: Ron Gonzalez
Assignee: Reuven Lax


Description Resource Path Location Type
The declared package "org.apache.beam.examples" does not match the expected 
package "org.apache.beam.examples.website_snippets" Snippets.java 
/beam-examples-java8/src/main/java/org/apache/beam/examples/website_snippets 
line 18 Java Problem

 

The problem is that Snippets.java is located in website_snippets directory, but 
the package name is org.apache.beam.examples only (without website_snippets).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #35

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8703343982784522947.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4339178226291347739.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6813816319471502116.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3597281251989641951.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3625556071116688340.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1696039923357114734.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #27

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4885987327756102663.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins8227554714921536369.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1767067257191007368.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7267651542699423208.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2177964909282513955.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_Spark #1247

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins855444339120892469.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7242045318419062220.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4180914772388266962.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5253680870536328637.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7210135944067700723.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2640107148516320186.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 

Build failed in Jenkins: beam_PerformanceTests_Python #803

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1751550996273696613.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2163746126221078473.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins735067987687937574.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3065146642491944025.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7545760625304875967.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #27

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4341930541604466518.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1047236098604865811.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6753500013068406955.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5697813720559110979.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2606307713251989797.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #28

[Apache Jenkins Server]

See 


Changes:

[niemo] Update BigTableSource to use multiple key ranges

[niemo] Add tests for supporting multiple key ranges.

[niemo] Use multiple key ranges in BigtableServiceImpl

[tgroh] Add wire_coder_id to RemoteGrpcPort

[tgroh] Add joda-time dependency to java/fn-execution

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 40c9081d4152462630558b12faa39d62a640d030 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 40c9081d4152462630558b12faa39d62a640d030
Commit message: "Merge pull request #4386"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins7534062366980216829.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3192677107729971258.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5081790252456046179.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5939165068504870390.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5852421927566022820.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

[jira] [Commented] (BEAM-3490) Reasonable Python direct runner batch performance.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329763#comment-16329763
 ] 

ASF GitHub Bot commented on BEAM-3490:
--

robertwb opened a new pull request #4437: [BEAM-3490] Curry CombineFn arguments 
into runner API protos.
URL: https://github.com/apache/beam/pull/4437
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Reasonable Python direct runner batch performance.
> --
>
> Key: BEAM-3490
> URL: https://issues.apache.org/jira/browse/BEAM-3490
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
>
> The plan is to migrate to the FnApi Runner for batch workloads. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3490) Reasonable Python direct runner batch performance.

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-3490:
-

 Summary: Reasonable Python direct runner batch performance.
 Key: BEAM-3490
 URL: https://issues.apache.org/jira/browse/BEAM-3490
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw


The plan is to migrate to the FnApi Runner for batch workloads. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3160) Type based coder inference incorrectly assumes that a coder for one type is equivalent to every other coder for that type.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329756#comment-16329756
 ] 

ASF GitHub Bot commented on BEAM-3160:
--

lukecwik opened a new pull request #4433: [BEAM-3160] Prevent issue where we 
would choose which coder to use arbitrarily when it is over specified with 
multiple coders.
URL: https://github.com/apache/beam/pull/4433
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Type based coder inference incorrectly assumes that a coder for one type is 
> equivalent to every other coder for that type.
> --
>
> Key: BEAM-3160
> URL: https://issues.apache.org/jira/browse/BEAM-3160
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.3.0
>
>
> We should prevent coder inference from assuming that two coders for the same 
> type are interchangeable.
> Two Avro GenericRecord coders with different schemas are considered identical 
> and an arbitrary one is returned by the Coder/Type inference system if the 
> GenericRecord type appears multiple times.
> e.g.
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaA)), 
> IterableCoder.of(AvroCoder.of(SchemaB)))* after coder inference for the type 
> *KV* will return 
> *KvCoder.of(IterableCoder.of(AvroCoder.of(SchemaX)), 
> IterableCoder.of(AvroCoder.of(SchemaX)))* where SchemaX is either SchemaA or 
> SchemaB.
> Code:
> https://github.com/apache/beam/blob/v2.1.1/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L420
>  and other Type -> Coder maps in the same file should prevent insertion if 
> the type already exists and the coders aren't equal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3450) RemoteGrpcPorts should contain the wire format

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329694#comment-16329694
 ] 

ASF GitHub Bot commented on BEAM-3450:
--

tgroh closed pull request #4386: [BEAM-3450] Add wire_coder_id to RemoteGrpcPort
URL: https://github.com/apache/beam/pull/4386
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto 
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index 36ed4242d6c..16f7709982d 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -76,6 +76,9 @@ message RemoteGrpcPort {
   // (Required) An API descriptor which describes where to
   // connect to including any authentication that is required.
   org.apache.beam.model.pipeline.v1.ApiServiceDescriptor 
api_service_descriptor = 1;
+
+  // (Required) The ID of the Coder that will be used to encode and decode 
data sent over this port.
+  string coder_id = 2;
 }
 
 /*
diff --git a/sdks/java/fn-execution/pom.xml b/sdks/java/fn-execution/pom.xml
index b20e72b62e3..ca520c87d52 100644
--- a/sdks/java/fn-execution/pom.xml
+++ b/sdks/java/fn-execution/pom.xml
@@ -66,6 +66,11 @@
   beam-sdks-java-core
 
 
+
+  joda-time
+  joda-time
+
+
 
   com.google.protobuf
   protobuf-java
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
index 3e9d8a01eba..f5db80a64c8 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
@@ -34,8 +34,10 @@
 import org.apache.beam.fn.harness.fn.ThrowingRunnable;
 import org.apache.beam.fn.harness.state.BeamFnStateClient;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort;
 import org.apache.beam.model.pipeline.v1.Endpoints;
 import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
 import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
 import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
 import org.apache.beam.runners.core.construction.CoderTranslation;
@@ -121,7 +123,7 @@
   private final Endpoints.ApiServiceDescriptor apiServiceDescriptor;
   private final FnDataReceiver receiver;
   private final Supplier processBundleInstructionIdSupplier;
-  private final BeamFnDataClient beamFnDataClientFactory;
+  private final BeamFnDataClient beamFnDataClient;
   private final Coder coder;
   private final BeamFnApi.Target inputTarget;
 
@@ -133,28 +135,37 @@
   BeamFnApi.Target inputTarget,
   RunnerApi.Coder coderSpec,
   Map coders,
-  BeamFnDataClient beamFnDataClientFactory,
+  BeamFnDataClient beamFnDataClient,
   Collection> consumers)
-  throws IOException {
-this.apiServiceDescriptor =
-
RemoteGrpcPortRead.fromPTransform(grpcReadNode).getPort().getApiServiceDescriptor();
+  throws IOException {
+RemoteGrpcPort port = 
RemoteGrpcPortRead.fromPTransform(grpcReadNode).getPort();
+this.apiServiceDescriptor = port.getApiServiceDescriptor();
 this.inputTarget = inputTarget;
 this.processBundleInstructionIdSupplier = 
processBundleInstructionIdSupplier;
-this.beamFnDataClientFactory = beamFnDataClientFactory;
+this.beamFnDataClient = beamFnDataClient;
 this.receiver = MultiplexingFnDataReceiver.forConsumers(consumers);
 
+RehydratedComponents components =
+
RehydratedComponents.forComponents(Components.newBuilder().putAllCoders(coders).build());
 @SuppressWarnings("unchecked")
-Coder coder =
-(Coder)
-CoderTranslation.fromProto(
-coderSpec,
-RehydratedComponents.forComponents(
-
RunnerApi.Components.newBuilder().putAllCoders(coders).build()));
+Coder coder;
+if (!port.getCoderId().isEmpty()) {
+  coder =
+  (Coder)
+  CoderTranslation.fromProto(coders.get(port.getCoderId()), 
components);
+} else {
+  // TODO: Remove this path once it is no longer used
+  coder =
+  (Coder)
+  CoderTranslation.fromProto(
+  coderSpec,
+  components);
+}
 this.coder = coder;
   }
 
   public void 

[beam] 01/01: Merge pull request #4386

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 40c9081d4152462630558b12faa39d62a640d030
Merge: dacd1e4 62e2a77
Author: Thomas Groh 
AuthorDate: Wed Jan 17 15:35:03 2018 -0800

Merge pull request #4386

[BEAM-3450] Add wire_coder_id to RemoteGrpcPort

 .../fn-execution/src/main/proto/beam_fn_api.proto  |  3 ++
 sdks/java/fn-execution/pom.xml |  5 +++
 .../beam/fn/harness/BeamFnDataReadRunner.java  | 37 +---
 .../beam/fn/harness/BeamFnDataWriteRunner.java | 23 -
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  | 16 ++---
 .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 40 ++
 6 files changed, 83 insertions(+), 41 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch master updated (dacd1e4 -> 40c9081)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from dacd1e4  Merge pull request #4312 from rniemo-g/multiple-row-ranges
 add 6a58c47  Add wire_coder_id to RemoteGrpcPort
 add 62e2a77  Add joda-time dependency to java/fn-execution
 new 40c9081  Merge pull request #4386

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fn-execution/src/main/proto/beam_fn_api.proto  |  3 ++
 sdks/java/fn-execution/pom.xml |  5 +++
 .../beam/fn/harness/BeamFnDataReadRunner.java  | 37 +---
 .../beam/fn/harness/BeamFnDataWriteRunner.java | 23 -
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  | 16 ++---
 .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 40 ++
 6 files changed, 83 insertions(+), 41 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Updated] (BEAM-3046) De-javafy the Python Metrics API

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada updated BEAM-3046:

Labels: community-onboarding starter  (was: community-onboarding)

> De-javafy the Python Metrics API
> 
>
> Key: BEAM-3046
> URL: https://issues.apache.org/jira/browse/BEAM-3046
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Pablo Estrada
>Priority: Major
>  Labels: community-onboarding, starter
>
> For example, though Java forces one to put all stand-alone functions as 
> static methods on a class (like Metrics) in Python one would just have 
> standard module-level functions. 
> Also, the MetricsFilter builder pattern is very Java-esque (and verbose). In 
> Python, one would simply let result.metrics() take keyword arguments. 
> We should seriously consider aggregating rather than returning lists of 
> results, at least by default. E.g. in 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py
>  it assumes that there's no other metric with the given name in any 
> subtransform (otherwise it may deterministically grab the "wrong" one.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3250:
-

Assignee: Ben Sidhom  (was: Kenneth Knowles)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Ben Sidhom
>Priority: Major
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3154) Support multiple KeyRanges when reading from BigTable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329498#comment-16329498
 ] 

ASF GitHub Bot commented on BEAM-3154:
--

jkff closed pull request #4312: [BEAM-3154] Support Multiple KeyRanges when 
reading from BigTable
URL: https://github.com/apache/beam/pull/4312
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 4199b28833c..71c041556a3 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -29,6 +29,7 @@
 import com.google.bigtable.v2.SampleRowKeysResponse;
 import com.google.cloud.bigtable.config.BigtableOptions;
 import com.google.common.base.MoreObjects;
+import com.google.common.base.MoreObjects.ToStringHelper;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
 import com.google.common.util.concurrent.FutureCallback;
@@ -36,6 +37,7 @@
 import com.google.common.util.concurrent.MoreExecutors;
 import com.google.protobuf.ByteString;
 import java.io.IOException;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.Iterator;
 import java.util.List;
@@ -77,8 +79,8 @@
  *
  * To configure a Cloud Bigtable source, you must supply a table id, a 
project id, an instance
  * id and optionally a {@link BigtableOptions} to provide more specific 
connection configuration.
- * By default, {@link BigtableIO.Read} will read all rows in the table. The 
row range to be read
- * can optionally be restricted using {@link BigtableIO.Read#withKeyRange}, 
and a {@link RowFilter}
+ * By default, {@link BigtableIO.Read} will read all rows in the table. The 
row ranges to be read
+ * can optionally be restricted using {@link BigtableIO.Read#withKeyRanges}, 
and a {@link RowFilter}
  * can be specified using {@link BigtableIO.Read#withRowFilter}. For example:
  *
  * {@code
@@ -189,7 +191,7 @@ public static Write write() {
 
 /** Returns the range of keys that will be read from the table. */
 @Nullable
-public abstract ByteKeyRange getKeyRange();
+public abstract List getKeyRanges();
 
 /** Returns the table being read from. */
 @Nullable
@@ -217,7 +219,7 @@ static Read create() {
 
   return new AutoValue_BigtableIO_Read.Builder()
 .setBigtableConfig(config)
-.setKeyRange(ByteKeyRange.ALL_KEYS)
+.setKeyRanges(Arrays.asList(ByteKeyRange.ALL_KEYS))
 .build();
 }
 
@@ -228,7 +230,7 @@ static Read create() {
 
   abstract Builder setRowFilter(RowFilter filter);
 
-  abstract Builder setKeyRange(ByteKeyRange keyRange);
+  abstract Builder setKeyRanges(List keyRange);
 
   abstract Read build();
 }
@@ -334,7 +336,22 @@ public Read withRowFilter(RowFilter filter) {
  */
 public Read withKeyRange(ByteKeyRange keyRange) {
   checkArgument(keyRange != null, "keyRange can not be null");
-  return toBuilder().setKeyRange(keyRange).build();
+  return toBuilder().setKeyRanges(Arrays.asList(keyRange)).build();
+}
+
+/**
+ * Returns a new {@link BigtableIO.Read} that will read only rows in the 
specified ranges.
+ * Ranges must not overlap.
+ *
+ * Does not modify this object.
+ */
+public Read withKeyRanges(List keyRanges) {
+  checkArgument(keyRanges != null, "keyRanges can not be null");
+  checkArgument(!keyRanges.isEmpty(), "keyRanges can not be empty");
+  for (ByteKeyRange range : keyRanges) {
+checkArgument(range != null, "keyRanges cannot hold null range");
+  }
+  return toBuilder().setKeyRanges(keyRanges).build();
 }
 
 /**
@@ -376,7 +393,7 @@ Read withBigtableService(BigtableService bigtableService) {
 public BigtableService apply(PipelineOptions options) {
   return getBigtableConfig().getBigtableService(options);
 }
-  }, getTableId(), getRowFilter(), getKeyRange(), null);
+  }, getTableId(), getRowFilter(), getKeyRanges(), null);
   return 
input.getPipeline().apply(org.apache.beam.sdk.io.Read.from(source));
 }
 
@@ -390,8 +407,11 @@ public void populateDisplayData(DisplayData.Builder 
builder) {
   super.populateDisplayData(builder);
   getBigtableConfig().populateDisplayData(builder);
 
-  builder.addIfNotDefault(
-  DisplayData.item("keyRange", getKeyRange().toString()), 

[beam] branch master updated (4da548e -> dacd1e4)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4da548e  Merge pull request #4224
 add a139fe8  Update BigTableSource to use multiple key ranges
 add 6131ca5  Add tests for supporting multiple key ranges.
 add da0ec24  Use multiple key ranges in BigtableServiceImpl
 add d0a2a02  Merge branch 'master' of https://github.com/apache/beam into 
multiple-row-ranges
 new dacd1e4  Merge pull request #4312 from rniemo-g/multiple-row-ranges

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   | 126 +--
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  18 ++-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   | 173 +++--
 3 files changed, 255 insertions(+), 62 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/01: Merge pull request #4312 from rniemo-g/multiple-row-ranges

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dacd1e41df9bb720fc3266624c5d50c82d83dc6c
Merge: 4da548e d0a2a02
Author: Eugene Kirpichov 
AuthorDate: Wed Jan 17 13:23:42 2018 -0800

Merge pull request #4312 from rniemo-g/multiple-row-ranges

[BEAM-3154] Support Multiple KeyRanges when reading from BigTable

 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   | 126 +--
 .../sdk/io/gcp/bigtable/BigtableServiceImpl.java   |  18 ++-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   | 173 +++--
 3 files changed, 255 insertions(+), 62 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4736

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3489) Expose the message id of received messages within PubsubMessage

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3489:

Issue Type: New Feature  (was: Bug)

> Expose the message id of received messages within PubsubMessage
> ---
>
> Key: BEAM-3489
> URL: https://issues.apache.org/jira/browse/BEAM-3489
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Priority: Minor
>  Labels: newbie, starter
>
> This task is about passing forward the message id from the pubsub proto to 
> the java PubsubMessage.
> Add a message id field to PubsubMessage.
> Update the coder for PubsubMessage to encode the message id.
> Update the translation from the Pubsub proto message to the Dataflow message:
> https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (2390f35 -> 7da45a1)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 2390f35  This closes #372
 add 7da45a1  Prepare repository for deployment.

No new revisions were added by this update.

Summary of changes:
 content/js/language-switch.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[jira] [Created] (BEAM-3489) Expose the message id of received messages within PubsubMessage

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-3489:
---

 Summary: Expose the message id of received messages within 
PubsubMessage
 Key: BEAM-3489
 URL: https://issues.apache.org/jira/browse/BEAM-3489
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Luke Cwik


This task is about passing forward the message id from the pubsub proto to the 
java PubsubMessage.

Add a message id field to PubsubMessage.

Update the coder for PubsubMessage to encode the message id.

Update the translation from the Pubsub proto message to the Dataflow message:

https://github.com/apache/beam/blob/2e275264b21db45787833502e5e42907b05e28b8/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L976



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (45d0dc0 -> 7da45a1)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 45d0dc0  Prepare repository for deployment.
 add 8a62de4  fix language tabs after a language specific paragraph
 add 2390f35  This closes #372
 new 7da45a1  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/js/language-switch.js | 2 +-
 src/js/language-switch.js | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7da45a1b7a86e96264b213a927c5fd8a9e2dc10c
Author: Mergebot 
AuthorDate: Wed Jan 17 18:50:07 2018 +

Prepare repository for deployment.
---
 content/js/language-switch.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/content/js/language-switch.js b/content/js/language-switch.js
index d183f81..24ddba2 100644
--- a/content/js/language-switch.js
+++ b/content/js/language-switch.js
@@ -48,7 +48,7 @@ $(document).ready(function() {
 var _self = this;
 
 $("div"+_self.selector).each(function() {
-if ($(this).prev().is(_self.selector)) {
+if ($(this).prev().is("div"+_self.selector)) {
 return;
 }
 $(this).before(_self.navHtml(_self.lookup($(this), [])));

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (ddad4fe -> 2390f35)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from ddad4fe  This closes #375
 add 45d0dc0  Prepare repository for deployment.
 new 8a62de4  fix language tabs after a language specific paragraph
 new 2390f35  This closes #372

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/presentation-materials/index.html | 2 +-
 content/contribute/website-contributions/index.html  | 2 +-
 content/contribute/work-in-progress/index.html   | 8 
 src/js/language-switch.js| 2 +-
 4 files changed, 7 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 02/02: This closes #372

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 2390f3537354235855b1353e2c9aed5099960bf5
Merge: 45d0dc0 8a62de4
Author: Mergebot 
AuthorDate: Wed Jan 17 18:42:41 2018 +

This closes #372

 src/js/language-switch.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/02: fix language tabs after a language specific paragraph

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8a62de44e0f118e683a20c4fe293ee82f03ce251
Author: melissa 
AuthorDate: Thu Jan 11 15:12:27 2018 -0800

fix language tabs after a language specific paragraph
---
 src/js/language-switch.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/js/language-switch.js b/src/js/language-switch.js
index d183f81..24ddba2 100644
--- a/src/js/language-switch.js
+++ b/src/js/language-switch.js
@@ -48,7 +48,7 @@ $(document).ready(function() {
 var _self = this;
 
 $("div"+_self.selector).each(function() {
-if ($(this).prev().is(_self.selector)) {
+if ($(this).prev().is("div"+_self.selector)) {
 return;
 }
 $(this).before(_self.navHtml(_self.lookup($(this), [])));

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 45d0dc00a55e08075be81c5b95f4259d1644c584
Author: Mergebot 
AuthorDate: Wed Jan 17 18:42:14 2018 +

Prepare repository for deployment.
---
 content/contribute/presentation-materials/index.html | 2 +-
 content/contribute/website-contributions/index.html  | 2 +-
 content/contribute/work-in-progress/index.html   | 8 
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/content/contribute/presentation-materials/index.html 
b/content/contribute/presentation-materials/index.html
index 8b5a641..b15f853 100644
--- a/content/contribute/presentation-materials/index.html
+++ b/content/contribute/presentation-materials/index.html
@@ -174,7 +174,7 @@
 Contributing new content
 The Apache Beam community is always looking for new core content and 
contributions!
 
-If you are interested in contributing new content, we recommend you first 
search the https://issues.apache.org/jira/browse/BEAM;>Apache Beam 
JIRA to see if your idea currently has an open JIRA item. If it does not, 
send an email to the mailto:u...@beam.apache.org;>u...@beam.apache.org mailing list or open 
a new issue in the https://issues.apache.org/jira/browse/BEAM;>Apache 
Beam JIRA using the https://issues.apache.org/jira/browse/BEAM;>Apache Beam 
JIRA to see if your idea currently has an open JIRA item. If it does not, 
send an email to the mailto:u...@beam.apache.org;>u...@beam.apache.org mailing list or open 
a new issue in the https://issues.apache.org/jira/browse/BEAM;>Apache 
Beam JIRA using the Corrections and updates
 If you encounter missing, incorrect, or out of date material, please make a 
comment in the presentation or send an email to the mailto:u...@beam.apache.org;>u...@beam.apache.org mailing list.
diff --git a/content/contribute/website-contributions/index.html 
b/content/contribute/website-contributions/index.html
index 41babb8..9edb853 100644
--- a/content/contribute/website-contributions/index.html
+++ b/content/contribute/website-contributions/index.html
@@ -165,7 +165,7 @@ mirror repository in the asf-site
 branch (not master).
 
 Issues are tracked in the
-https://issues.apache.org/jira/browse/BEAM/component/12328906;>website
+https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website;>website
 component in JIRA.
 
 One-time Setup
diff --git a/content/contribute/work-in-progress/index.html 
b/content/contribute/work-in-progress/index.html
index b901f80..48c7a36 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -191,25 +191,25 @@
 
   JStorm Runner
   https://github.com/apache/beam/tree/jstorm-runner;>jstorm-runner
-  https://issues.apache.org/jira/browse/BEAM/component/12332477;>runner-jstorm
+  https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-jstorm;>runner-jstorm
   https://issues.apache.org/jira/browse/BEAM-1899;>BEAM-1899
 
 
   MapReduce Runner
   https://github.com/apache/beam/tree/mr-runner;>mr-runner
-  https://issues.apache.org/jira/browse/BEAM/component/12333013;>runner-mapreduce
+  https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-mapreduce;>runner-mapreduce
   https://issues.apache.org/jira/browse/BEAM-165;>BEAM-165
 
 
   Tez Runner
   https://github.com/apache/beam/tree/tez-runner;>tez-runner
-  https://issues.apache.org/jira/browse/BEAM/component/12333014;>runner-tez
+  https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-tez;>runner-tez
   https://issues.apache.org/jira/browse/BEAM-2709;>BEAM-2709
 
 
   Go SDK
   https://github.com/apache/beam/tree/go-sdk;>go-sdk
-  https://issues.apache.org/jira/browse/BEAM/component/12333564;>sdk-go
+  https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20sdk-go;>sdk-go
   https://issues.apache.org/jira/browse/BEAM-2083;>BEAM-2083
 
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (79e62e2 -> 45d0dc0)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 79e62e2  Prepare repository for deployment.
 add 2650eec  Fix broken JIRA component links
 add ddad4fe  This closes #375
 new 45d0dc0  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/presentation-materials/index.html | 2 +-
 content/contribute/website-contributions/index.html  | 2 +-
 content/contribute/work-in-progress/index.html   | 8 
 src/contribute/presentation-materials.md | 2 +-
 src/contribute/website-contributions.md  | 2 +-
 src/contribute/work-in-progress.md   | 8 
 6 files changed, 12 insertions(+), 12 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] branch mergebot updated (7f22a3f -> ddad4fe)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 7f22a3f  This closes #375
 new ddad4fe  This closes #375

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7f22a3f)
\
 N -- N -- N   refs/heads/mergebot (ddad4fe)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: This closes #375

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ddad4fec2962f2e5a3ec29112aedb709fa10f1b9
Merge: 79e62e2 2650eec
Author: Mergebot 
AuthorDate: Wed Jan 17 18:26:57 2018 +

This closes #375

 src/contribute/presentation-materials.md | 2 +-
 src/contribute/website-contributions.md  | 2 +-
 src/contribute/work-in-progress.md   | 8 
 3 files changed, 6 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is back to normal : beam_PerformanceTests_Spark #1246

[Apache Jenkins Server]

See 

[beam-site] 02/02: This closes #375

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 7f22a3f522c47948be2d8cdd508261afde2a527f
Merge: 79e62e2 2650eec
Author: Mergebot 
AuthorDate: Wed Jan 17 18:14:03 2018 +

This closes #375

 src/contribute/presentation-materials.md | 2 +-
 src/contribute/website-contributions.md  | 2 +-
 src/contribute/work-in-progress.md   | 8 
 3 files changed, 6 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (a44b123 -> 7f22a3f)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from a44b123  This closes #373
 add 79e62e2  Prepare repository for deployment.
 new 2650eec  Fix broken JIRA component links
 new 7f22a3f  This closes #375

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/docker-images/index.html| 11 ++-
 content/get-started/quickstart-java/index.html |  2 +-
 src/contribute/presentation-materials.md   |  2 +-
 src/contribute/website-contributions.md|  2 +-
 src/contribute/work-in-progress.md |  8 
 5 files changed, 9 insertions(+), 16 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: Fix broken JIRA component links

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 2650eece833ca1bc7787ba3a69c11eb2452a3ac7
Author: melissa 
AuthorDate: Tue Jan 16 12:55:12 2018 -0800

Fix broken JIRA component links
---
 src/contribute/presentation-materials.md | 2 +-
 src/contribute/website-contributions.md  | 2 +-
 src/contribute/work-in-progress.md   | 8 
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/contribute/presentation-materials.md 
b/src/contribute/presentation-materials.md
index 93ce174..cac43a6 100644
--- a/src/contribute/presentation-materials.md
+++ b/src/contribute/presentation-materials.md
@@ -26,7 +26,7 @@ The Apache Beam community is looking for feedback on these 
materials to improve
 ## Contributing new content
 The Apache Beam community is always looking for new core content and 
contributions!
 
-If you are interested in contributing new content, we recommend you first 
search the [Apache Beam JIRA](https://issues.apache.org/jira/browse/BEAM) to 
see if your idea currently has an open JIRA item. If it does not, send an email 
to the [`u...@beam.apache.org`](mailto:u...@beam.apache.org) mailing list or 
open a new issue in the [Apache Beam 
JIRA](https://issues.apache.org/jira/browse/BEAM) using the 
[website](https://issues.apache.org/jira/browse/BEAM/component/12328906/?selectedTab=co
 [...]
+If you are interested in contributing new content, we recommend you first 
search the [Apache Beam JIRA](https://issues.apache.org/jira/browse/BEAM) to 
see if your idea currently has an open JIRA item. If it does not, send an email 
to the [`u...@beam.apache.org`](mailto:u...@beam.apache.org) mailing list or 
open a new issue in the [Apache Beam 
JIRA](https://issues.apache.org/jira/browse/BEAM) using the 
[website](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20compo
 [...]
 
 ## Corrections and updates
 If you encounter missing, incorrect, or out of date material, please make a 
comment in the presentation or send an email to the 
[`u...@beam.apache.org`](mailto:u...@beam.apache.org) mailing list.
diff --git a/src/contribute/website-contributions.md 
b/src/contribute/website-contributions.md
index e46b498..dc3c56f 100644
--- a/src/contribute/website-contributions.md
+++ b/src/contribute/website-contributions.md
@@ -16,7 +16,7 @@ mirror](https://github.com/apache/beam-site) repository in 
the `asf-site`
 branch (_not_ `master`).
 
 Issues are tracked in the
-[website](https://issues.apache.org/jira/browse/BEAM/component/12328906)
+[website](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20website)
 component in JIRA.
 
 ## One-time Setup
diff --git a/src/contribute/work-in-progress.md 
b/src/contribute/work-in-progress.md
index c9ebd4e..1c8eb5f 100644
--- a/src/contribute/work-in-progress.md
+++ b/src/contribute/work-in-progress.md
@@ -26,8 +26,8 @@ Current branches include:
 | Feature | Branch | JIRA Component | More Info |
 |  |  |  |  |
 | Apache Spark 2.0 Runner | 
[runners-spark2](https://github.com/apache/beam/tree/runners-spark2) | - | 
[thread](https://lists.apache.org/thread.html/e38ac4e4914a6cb1b865b1f32a6ca06c2be28ea4aa0f6b18393de66f@%3Cdev.beam.apache.org%3E)
 |
-| [JStorm Runner]({{ site.baseurl }}/documentation/runners/jstorm) | 
[jstorm-runner](https://github.com/apache/beam/tree/jstorm-runner) | 
[runner-jstorm](https://issues.apache.org/jira/browse/BEAM/component/12332477) 
| [BEAM-1899](https://issues.apache.org/jira/browse/BEAM-1899) |
-| MapReduce Runner | 
[mr-runner](https://github.com/apache/beam/tree/mr-runner) | 
[runner-mapreduce](https://issues.apache.org/jira/browse/BEAM/component/12333013)
 | [BEAM-165](https://issues.apache.org/jira/browse/BEAM-165) |
-| Tez Runner | [tez-runner](https://github.com/apache/beam/tree/tez-runner) | 
[runner-tez](https://issues.apache.org/jira/browse/BEAM/component/12333014) | 
[BEAM-2709](https://issues.apache.org/jira/browse/BEAM-2709) |
-| Go SDK | [go-sdk](https://github.com/apache/beam/tree/go-sdk) | 
[sdk-go](https://issues.apache.org/jira/browse/BEAM/component/12333564) | 
[BEAM-2083](https://issues.apache.org/jira/browse/BEAM-2083) |
+| [JStorm Runner]({{ site.baseurl }}/documentation/runners/jstorm) | 
[jstorm-runner](https://github.com/apache/beam/tree/jstorm-runner) | 
[runner-jstorm](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-jstorm)
 | [BEAM-1899](https://issues.apache.org/jira/browse/BEAM-1899) |
+| MapReduce Runner | 
[mr-runner](https://github.com/apache/beam/tree/mr-runner) | 
[runner-mapreduce](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-mapreduce)
 | [BEAM-165](https://issues.apache.org/jira/browse/BEAM-165) |
+| Tez Runner | 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #26

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 >  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6405343393986161243.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4013291289931944197.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7360975471174156534.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3197547531628714295.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3361226269852376010.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 

Build failed in Jenkins: beam_PerformanceTests_Python #802

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4714855682459180193.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5948525854507735433.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins213902006961179364.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5621201568096929840.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins844557816106572186.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1330450193949001669.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #34

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2820704787881494702.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4031932069224663599.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8388869526729339082.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6361819686278302341.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8978136976375400983.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins968099890439657615.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #26

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3351548928225595693.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4012015582052477729.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4948549921436959753.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1807395219051418927.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins205148170597438407.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1744027098937561074.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #27

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list 4da548ec2191c25c4524b5a1076e09f18aeb4446 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3999607005702580023.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5369293248761892881.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6874214134453078334.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins67920025978863985.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4979948169192085554.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5756224219968376718.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

[beam] 04/04: Add initialization of active plans map.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c5a3ce00eeb96ba4e357080d6bd27708273da295
Author: Bill Neubauer 
AuthorDate: Wed Jan 17 09:43:24 2018 -0800

Add initialization of active plans map.
---
 sdks/go/pkg/beam/core/runtime/harness/harness.go | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go 
b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 8822c62..1dd0d2e 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -85,8 +85,9 @@ func Main(ctx context.Context, loggingEndpoint, 
controlEndpoint string) error {
}()
 
ctrl := {
-   plans: make(map[string]*exec.Plan),
-   data:  {},
+   plans:  make(map[string]*exec.Plan),
+   active: make(map[string]*exec.Plan),
+   data:   {},
}
 
var cpuProfBuf bytes.Buffer

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] 02/04: Remove GetId() call from under lock.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a86c25e372c39e6a6780fa262ad45105a46e00b9
Author: Bill Neubauer 
AuthorDate: Tue Jan 16 14:29:08 2018 -0800

Remove GetId() call from under lock.

Don't want to have anything happening while holding the plan
lock that isn't obvious it can't fail.
---
 sdks/go/pkg/beam/core/runtime/harness/harness.go | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go 
b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 72db1f1..335e2b3 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -176,10 +176,12 @@ func (c *control) handleInstruction(ctx context.Context, 
req *fnpb.InstructionRe
if err != nil {
return fail(id, "translation failed: %v", err)
}
-   log.Debugf(ctx, "Plan %v: %v", desc.GetId(), plan)
+
+   pid := desc.GetId()
+   log.Debugf(ctx, "Plan %v: %v", pid, plan)
 
c.mu.Lock()
-   c.plans[desc.GetId()] = plan
+   c.plans[pid] = plan
c.mu.Unlock()
}
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam] branch go-sdk updated (864b359 -> c5a3ce0)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 864b359  BEAM-3474 Include stacks in panic messages.
 new 0af2503  BEAM-3299: Add source reporting support.
 new a86c25e  Remove GetId() call from under lock.
 new efa1c20  Add additional comments about concurrency invariants.
 new c5a3ce0  Add initialization of active plans map.

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/exec/datasource.go |  25 +
 sdks/go/pkg/beam/core/runtime/exec/plan.go   |  12 +++
 sdks/go/pkg/beam/core/runtime/harness/harness.go | 117 +--
 3 files changed, 125 insertions(+), 29 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam] 01/04: BEAM-3299: Add source reporting support.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0af2503d72c20dd88b9e63a557c63319068d70af
Author: Bill Neubauer 
AuthorDate: Sun Jan 14 20:15:13 2018 -0800

BEAM-3299: Add source reporting support.

Makes the harness sufficiently multithreaded so we can handle progress
reports while we are performing work. The plans maintaing their ability
to be reused, but still can't be used concurrently.
---
 sdks/go/pkg/beam/core/runtime/exec/datasource.go | 25 ++
 sdks/go/pkg/beam/core/runtime/exec/plan.go   | 12 +++
 sdks/go/pkg/beam/core/runtime/harness/harness.go | 97 ++--
 3 files changed, 109 insertions(+), 25 deletions(-)

diff --git a/sdks/go/pkg/beam/core/runtime/exec/datasource.go 
b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
index e29239b..6a87978 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/datasource.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
@@ -19,9 +19,12 @@ import (
"context"
"fmt"
"io"
+   "sync/atomic"
+   "time"
 
"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
 )
 
 // DataSource is a Root execution unit.
@@ -32,6 +35,8 @@ type DataSource struct {
 
sidStreamID
source DataReader
+   count  int64
+   start  time.Time
 }
 
 func (n *DataSource) ID() UnitID {
@@ -45,6 +50,8 @@ func (n *DataSource) Up(ctx context.Context) error {
 func (n *DataSource) StartBundle(ctx context.Context, id string, data 
DataManager) error {
n.sid = StreamID{Port: *n.Edge.Port, Target: *n.Edge.Target, InstID: id}
n.source = data
+   n.start = time.Now()
+   atomic.StoreInt64(, 0)
return n.Out.StartBundle(ctx, id, data)
 }
 
@@ -93,6 +100,7 @@ func (n *DataSource) Process(ctx context.Context) error {
// Single chunk stream.
 
// log.Printf("Fixed size=%v", size)
+   atomic.AddInt64(, int64(size))
 
for i := int32(0); i < size; i++ {
value, err := cv.Decode(r)
@@ -116,6 +124,7 @@ func (n *DataSource) Process(ctx context.Context) error {
break
}
 
+   atomic.AddInt64(, int64(chunk))
for i := uint64(0); i < chunk; i++ {
value, err := cv.Decode(r)
if err != nil {
@@ -139,6 +148,7 @@ func (n *DataSource) Process(ctx context.Context) error {
ec := MakeElementDecoder(coder.SkipW(c))
 
for {
+   atomic.AddInt64(, 1)
t, err := DecodeWindowedValueHeader(c, r)
if err != nil {
if err == io.EOF {
@@ -163,6 +173,7 @@ func (n *DataSource) Process(ctx context.Context) error {
 }
 
 func (n *DataSource) FinishBundle(ctx context.Context) error {
+   log.Infof(context.Background(), "DataSource: %d elements in %d ns", 
n.count, time.Now().Sub(n.start))
n.sid = StreamID{}
n.source = nil
return n.Out.FinishBundle(ctx)
@@ -178,3 +189,17 @@ func (n *DataSource) String() string {
sid := StreamID{Port: *n.Edge.Port, Target: *n.Edge.Target}
return fmt.Sprintf("DataSource[%v] Out:%v", sid, n.Out.ID())
 }
+
+// ProgressReportSnapshot captures the progress reading an input source.
+type ProgressReportSnapshot struct {
+   ID, Name string
+   Countint64
+}
+
+// Progress returns a snapshot of the source's progress.
+func (n *DataSource) Progress() ProgressReportSnapshot {
+   if n == nil {
+   return ProgressReportSnapshot{}
+   }
+   return ProgressReportSnapshot{n.sid.Target.ID, n.sid.Target.Name, 
atomic.LoadInt64()}
+}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/plan.go 
b/sdks/go/pkg/beam/core/runtime/exec/plan.go
index d26fb1f..732f315 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/plan.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/plan.go
@@ -32,11 +32,14 @@ type Plan struct {
units []Unit
 
status Status
+   source *DataSource
 }
 
 // NewPlan returns a new bundle execution plan from the given units.
 func NewPlan(id string, units []Unit) (*Plan, error) {
var roots []Root
+   var source *DataSource
+
for _, u := range units {
if u == nil {
return nil, fmt.Errorf("no  units")
@@ -44,6 +47,9 @@ func NewPlan(id string, units []Unit) (*Plan, error) {
if r, ok := u.(Root); ok {

[beam] 03/04: Add additional comments about concurrency invariants.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git

commit efa1c20a9328177a6cd6ec4d744853417fd0ef48
Author: Bill Neubauer 
AuthorDate: Tue Jan 16 15:39:20 2018 -0800

Add additional comments about concurrency invariants.

Explain how gRPC-specific goroutines are managed.
Explain how plans are managed in the active and plans maps.
---
 sdks/go/pkg/beam/core/runtime/harness/harness.go | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go 
b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 335e2b3..8822c62 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -90,6 +90,11 @@ func Main(ctx context.Context, loggingEndpoint, 
controlEndpoint string) error {
}
 
var cpuProfBuf bytes.Buffer
+
+   // gRPC requires all readers of a stream be the same goroutine, so this 
goroutine
+   // is responsible for managing the network data. All it does is pull 
data from
+   // the stream, and hand off the message to a goroutine to actually be 
handled,
+   // so as to avoid blocking the underlying network channel.
for {
req, err := client.Recv()
if err != nil {
@@ -132,10 +137,14 @@ func Main(ctx context.Context, loggingEndpoint, 
controlEndpoint string) error {
 }
 
 type control struct {
-   plans  map[string]*exec.Plan // protected by mu
+   // plans that are candidates for execution.
+   plans map[string]*exec.Plan // protected by mu
+   // plans that are actively being executed.
+   // a plan can only be in one of these maps at any time.
active map[string]*exec.Plan // protected by mu
mu sync.Mutex
-   data   *DataManager
+
+   data *DataManager
 }
 
 func (c *control) handleInstruction(ctx context.Context, req 
*fnpb.InstructionRequest) *fnpb.InstructionResponse {

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-3488) Reduce log noise coming from File sink

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329091#comment-16329091
 ] 

Luke Cwik commented on BEAM-3488:
-

Its strange that it happens so many times since it should only happen once. I'm 
wondering whether a feature that was added like date sharded files is making it 
so that we output so much. How is your sink configured?

> Reduce log noise coming from File sink
> --
>
> Key: BEAM-3488
> URL: https://issues.apache.org/jira/browse/BEAM-3488
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Kenneth Knowles
>Priority: Minor
>
> After switching to Beam 2.2 I noticed that File sink related classes generate 
> lots of lines likes this:
> {code:java}
> 2018-01-16 01:37:37,080 INFO org.apache.beam.sdk.io.FileBasedSink - No output 
> files to write.
> 2018-01-16 01:37:37,104 INFO org.apache.beam.sdk.io.WriteFiles - Will 
> finalize 0 files{code}
>  
> I did some counts and it looks like this lines account for 82% of all lines 
> in the log yet not giving much information.
> I am happy to raise a PR to make "No output files to write." and "Will 
> finalize {} files" logged at DEBUG level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3285) Switch Beam to Java 8 only

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3285.

   Resolution: Duplicate
Fix Version/s: (was: 2.3.0)
   Not applicable

> Switch Beam to Java 8 only
> --
>
> Key: BEAM-3285
> URL: https://issues.apache.org/jira/browse/BEAM-3285
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: Not applicable
>
>
> See this thread 
> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E
>  
> The current JIRA is for follow-up items: decide when to conclude the vote, 
> and items on actually doing the transition (configuring pom.xml, modernizing 
> existing code etc)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3425) CassandraIO fails to estimate size: Codec not found for requested operation: [varchar <-> java.lang.Long]

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329051#comment-16329051
 ] 

ASF GitHub Bot commented on BEAM-3425:
--

jbonofre opened a new pull request #4426: [BEAM-3425] Get the range start & end 
as String
URL: https://github.com/apache/beam/pull/4426
 
 
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [X] Each commit in the pull request should have a meaningful subject line 
and body.
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
- [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> CassandraIO fails to estimate size: Codec not found for requested operation: 
> [varchar <-> java.lang.Long]
> -
>
> Key: BEAM-3425
> URL: https://issues.apache.org/jira/browse/BEAM-3425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> See exception in 
> https://stackoverflow.com/questions/48090668/how-to-increase-dataflow-read-parallelism-from-cassandra/48131264#48131264
>  .
> The exception comes from 
> https://github.com/apache/beam/blob/master/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java#L279
>  , where I suppose "range_start" and "range_end" are really varchar, but the 
> code expects them to be long.
> Indeed they are varchar: 
> https://github.com/apache/cassandra/blob/4c80eeece37d79f434078224a0504400ae10a20d/src/java/org/apache/cassandra/db/SystemKeyspace.java#L238
>  and have been for at least the past 3 years.
> However really they seem to be storing longs: 
> https://github.com/apache/cassandra/blob/95b43b195e4074533100f863344c182a118a8b6c/src/java/org/apache/cassandra/hadoop/cql3/CqlInputFormat.java#L229
> So I guess all that needs to be fixed is adding a Long.parseLong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-71) Watermark library

[Tarush Grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328934#comment-16328934
 ] 

Tarush Grover commented on BEAM-71:
---

[~lcwik] Thanks for assigning, I will document the interface first and get 
reviewed.

[~jkff] Thanks a lot for the example. Let me go through this and will come up 
with the design.

> Watermark library
> -
>
> Key: BEAM-71
> URL: https://issues.apache.org/jira/browse/BEAM-71
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Tarush Grover
>Priority: Major
>  Labels: Watermark
>
> See also: BEAM-70
> We should have a library of watermark implementations:
> * now: for when items use arrival timestamps
> * latest element: for when events arrive in timestamp order
> * moving average over last K time intervals
> * bucketed moving average, ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3488) Reduce log noise coming from File sink

[Pawel Bartoszek (JIRA)]

Pawel Bartoszek created BEAM-3488:
-

 Summary: Reduce log noise coming from File sink
 Key: BEAM-3488
 URL: https://issues.apache.org/jira/browse/BEAM-3488
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Affects Versions: 2.2.0
Reporter: Pawel Bartoszek
Assignee: Kenneth Knowles


After switching to Beam 2.2 I noticed that File sink related classes generate 
lots of lines likes this:
{code:java}
2018-01-16 01:37:37,080 INFO org.apache.beam.sdk.io.FileBasedSink - No output 
files to write.
2018-01-16 01:37:37,104 INFO org.apache.beam.sdk.io.WriteFiles - Will finalize 
0 files{code}
 

I did some counts and it looks like this lines account for 82% of all lines in 
the log yet not giving much information.

I am happy to raise a PR to make "No output files to write." and "Will finalize 
{} files" logged at DEBUG level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4735

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #801

[Apache Jenkins Server]

See 


Changes:

[kedin] [Nexmark][SQL] Implement Query1

[tgroh] Move BeamFnDataGrpcMultiplexer to java-fn-execution

[tgroh] Mark BeamFnDataGrpcMultiplexer ApiServiceDescriptor as Nullable

[tgroh] Enable Java8 for fn execution modules

[tgroh] Update GRPC Multiplexer and Associated to Java8

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list e977cdff1718b2316115f63903762380c15530de # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1137789986977891174.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7159714549836575533.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6958673806892171360.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins937354010526847486.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6423037473912671204.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #26

[Apache Jenkins Server]

See 


Changes:

[kedin] [Nexmark][SQL] Implement Query1

[tgroh] Move BeamFnDataGrpcMultiplexer to java-fn-execution

[tgroh] Mark BeamFnDataGrpcMultiplexer ApiServiceDescriptor as Nullable

[tgroh] Enable Java8 for fn execution modules

[tgroh] Update GRPC Multiplexer and Associated to Java8

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list e977cdff1718b2316115f63903762380c15530de # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins560770777072272.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4673237379745169382.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4760813500741527583.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4594255645483139187.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1035201763533352422.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Cleaning up...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2065543397846991925.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 

Build failed in Jenkins: beam_PerformanceTests_Spark #1245

[Apache Jenkins Server]

See 


Changes:

[kedin] [Nexmark][SQL] Implement Query1

[tgroh] Move BeamFnDataGrpcMultiplexer to java-fn-execution

[tgroh] Mark BeamFnDataGrpcMultiplexer ApiServiceDescriptor as Nullable

[tgroh] Enable Java8 for fn execution modules

[tgroh] Update GRPC Multiplexer and Associated to Java8

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4da548ec2191c25c4524b5a1076e09f18aeb4446 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4da548ec2191c25c4524b5a1076e09f18aeb4446
Commit message: "Merge pull request #4224"
 > git rev-list e977cdff1718b2316115f63903762380c15530de # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7309087597961528584.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3649622398984690219.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5983674672227662729.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4466776688742075406.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5014388350493272675.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4734

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3487) GroupByKey stalls with GroupingShuffleReader split refusals

[Bei Zhang (JIRA)]

Bei Zhang created BEAM-3487:
---

 Summary: GroupByKey stalls with GroupingShuffleReader split 
refusals
 Key: BEAM-3487
 URL: https://issues.apache.org/jira/browse/BEAM-3487
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Affects Versions: 2.2.0
Reporter: Bei Zhang
Assignee: Thomas Groh


With info messages with something like:
{quote}{{Refused to split GroupingShuffleReader  at 
ShufflePosition(base64:A1CWNvkAAQ)}}
{quote}
The lull messages look like this:
{quote}{{Processing lull for PT300.006S in state read-shuffle of Write 
Vectors2/GroupIntoShards/Read at 
com.google.cloud.dataflow.worker.ApplianceShuffleReader.readIncludingPosition(Native
 Method) at 
com.google.cloud.dataflow.worker.ChunkingShuffleBatchReader.read(ChunkingShuffleBatchReader.java:62)
 at 
com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:57)
 at 
com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader$1.load(CachingShuffleBatchReader.java:53)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
 at 
com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
 at 
com.google.cloud.dataflow.worker.util.common.worker.CachingShuffleBatchReader.read(CachingShuffleBatchReader.java:76)
 at 
com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntries(BatchingShuffleEntryReader.java:133)
 at 
com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.fillEntriesIfNeeded(BatchingShuffleEntryReader.java:126)
 at 
com.google.cloud.dataflow.worker.util.common.worker.BatchingShuffleEntryReader$ShuffleReadIterator.hasNext(BatchingShuffleEntryReader.java:90)
 at 
com.google.cloud.dataflow.worker.util.common.ForwardingReiterator.hasNext(ForwardingReiterator.java:62)
 at 
com.google.cloud.dataflow.worker.util.common.worker.GroupingShuffleEntryIterator.advance(GroupingShuffleEntryIterator.java:118)
 at 
com.google.cloud.dataflow.worker.GroupingShuffleReader$GroupingShuffleReaderIterator.advance(GroupingShuffleReader.java:230)
 at 
com.google.cloud.dataflow.worker.GroupingShuffleReader$GroupingShuffleReaderIterator.start(GroupingShuffleReader.java:224)
 at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:347)
 at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:183)
 at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:148)
 at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:68)
 at 
com.google.cloud.dataflow.worker.DataflowWorker.executeWork(DataflowWorker.java:330)
 at 
com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:302) 
at 
com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:251)
 at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
 at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
 at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)}}
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3153) Allow streaming processing time domain timers in Beam Python DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328473#comment-16328473
 ] 

ASF GitHub Bot commented on BEAM-3153:
--

mariapython opened a new pull request #4425: [BEAM-3153] Add test for 
processing-time timer
URL: https://github.com/apache/beam/pull/4425
 
 
   - [x] Add test for processing-time (real-time) timers 
   - [x] Add `class AfterProcessingTime`
   - [x] Advance processing time for `_TestStreamEvaluator`
   - [x] Re-engineer _should_shutdown (a realtime timer set no longer means 
_is_executing() should return True, it just makes _should_shutdown return False)
   - [x] Add test for watermark timers


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow streaming processing time domain timers in Beam Python DirectRunner
> -
>
> Key: BEAM-3153
> URL: https://issues.apache.org/jira/browse/BEAM-3153
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: María GH
>Priority: Major
>
> Currently, streaming execution in the Python Beam DirectRunner only allows 
> for watermark domain timers and does not allow for processing time timers.  
> We should support processing time timers, along with a testing framework 
> (potentially with a fake clock) that lets us test this behavior using the 
> TestStream primitive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)