[jira] [Updated] (BEAM-3235) Apache Beam with Apache Apex error

[ethanhunt999 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ethanhunt999 updated BEAM-3235:
---
Affects Version/s: 2.1.0

> Apache Beam with Apache Apex error
> --
>
> Key: BEAM-3235
> URL: https://issues.apache.org/jira/browse/BEAM-3235
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-apex
>Affects Versions: 2.1.0
>Reporter: ethanhunt999
>
> We are getting below error while running the Apache Beam with Apache Apex 
> Runner.
> java.lang.IllegalArgumentException: Cannot output with timestamp 
> 2017-11-21T07:27:46.431Z. Output timestamps must be no earlier than the 
> timestamp of the current input (2017-11-21T16:24:46.322Z) minus the allowed 
> skew (0 milliseconds).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3235) Apache Beam with Apache Apex error

[ethanhunt999 (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ethanhunt999 updated BEAM-3235:
---
Component/s: beam-model

> Apache Beam with Apache Apex error
> --
>
> Key: BEAM-3235
> URL: https://issues.apache.org/jira/browse/BEAM-3235
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, runner-apex
>Affects Versions: 2.1.0
>Reporter: ethanhunt999
>
> We are getting below error while running the Apache Beam with Apache Apex 
> Runner.
> java.lang.IllegalArgumentException: Cannot output with timestamp 
> 2017-11-21T07:27:46.431Z. Output timestamps must be no earlier than the 
> timestamp of the current input (2017-11-21T16:24:46.322Z) minus the allowed 
> skew (0 milliseconds).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3235) Apache Beam with Apache Apex error

[ethanhunt999 (JIRA)]

ethanhunt999 created BEAM-3235:
--

 Summary: Apache Beam with Apache Apex error
 Key: BEAM-3235
 URL: https://issues.apache.org/jira/browse/BEAM-3235
 Project: Beam
  Issue Type: Bug
  Components: runner-apex
Reporter: ethanhunt999


We are getting below error while running the Apache Beam with Apache Apex 
Runner.

java.lang.IllegalArgumentException: Cannot output with timestamp 
2017-11-21T07:27:46.431Z. Output timestamps must be no earlier than the 
timestamp of the current input (2017-11-21T16:24:46.322Z) minus the allowed 
skew (0 milliseconds).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #315

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3594

[Apache Jenkins Server]

See 


--
[...truncated 815.71 KB...]
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker

[2/2] beam git commit: Fix test runtime class conflicts and get hbase tests passing.

[lcwik]

Fix test runtime class conflicts and get hbase tests passing.

This closes #4166


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/558aa2fc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/558aa2fc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/558aa2fc

Branch: refs/heads/master
Commit: 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09
Parents: 4847d34 07d6607
Author: Luke Cwik 
Authored: Wed Nov 22 16:33:56 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 16:33:56 2017 -0800

--
 sdks/java/io/hbase/build.gradle | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)
--

[1/2] beam git commit: Fix test runtime class conflicts and get hbase tests passing.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 4847d34d7 -> 558aa2fc3


Fix test runtime class conflicts and get hbase tests passing.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/07d6607b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/07d6607b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/07d6607b

Branch: refs/heads/master
Commit: 07d6607b54110876ad081e3f680e88f5cff679f5
Parents: 4847d34
Author: Luke Cwik 
Authored: Wed Nov 22 10:50:43 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 16:33:34 2017 -0800

--
 sdks/java/io/hbase/build.gradle | 18 ++
 1 file changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/07d6607b/sdks/java/io/hbase/build.gradle
--
diff --git a/sdks/java/io/hbase/build.gradle b/sdks/java/io/hbase/build.gradle
index bcfebbd..dbfceb0 100644
--- a/sdks/java/io/hbase/build.gradle
+++ b/sdks/java/io/hbase/build.gradle
@@ -29,13 +29,6 @@ test {
   }
 }
 
-test {
-  // TODO: Get tests to run. Known issues:
-  //  * hbase-server:tests conflict with hbase-shaded-server with ProtobufUtil
-  ignoreFailures true
-}
-
-
 def hbase_version = "1.2.6"
 
 dependencies {
@@ -54,7 +47,16 @@ dependencies {
   testCompile library.java.hadoop_hdfs
   testCompile library.java.hadoop_common
   testCompile "org.apache.hbase:hbase-shaded-server:$hbase_version"
-  testCompile "org.apache.hbase:hbase-server:$hbase_version:tests"
+  testCompile("org.apache.hbase:hbase-server:$hbase_version:tests") {
+// We prevent bringing in unshaded dependencies to not conflict
+// with hbase-shaded-server and hbase-shaded-client
+transitive = false
+  }
+  testCompile("org.apache.hbase:hbase-common:$hbase_version:tests") {
+// We prevent bringing in unshaded dependencies to not conflict
+// with hbase-shaded-server and hbase-shaded-client
+transitive = false
+  }
   testCompile "org.apache.hbase:hbase-hadoop-compat:$hbase_version:tests"
   testCompile "org.apache.hbase:hbase-hadoop2-compat:$hbase_version:tests"
 }

[GitHub] beam pull request #4166: Fix test runtime class conflicts and get hbase test...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4166


---

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #321

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09
Commit message: "Fix test runtime class conflicts and get hbase tests passing."
 > git rev-list 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1112725795215083316.sh
+ bash sdks/python/run_validatesrunner.sh
bash: sdks/python/run_validatesrunner.sh: No such file or directory
Build step 'Execute shell' marked build as failure

[1/4] beam git commit: Move Common Fn Execution Concepts to fn-execution

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 5c74022da -> 4847d34d7


http://git-wip-us.apache.org/repos/asf/beam/blob/f9e2be91/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutors.java
--
diff --git 
a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutors.java
 
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutors.java
new file mode 100644
index 000..68f1583
--- /dev/null
+++ 
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutors.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.fn.test;
+
+import com.google.common.util.concurrent.ForwardingExecutorService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+/**
+ * A {@link TestRule} that validates that all submitted tasks finished and 
were completed. This
+ * allows for testing that tasks have exercised the appropriate shutdown logic.
+ */
+public class TestExecutors {
+  public static TestExecutorService from(final ExecutorService 
staticExecutorService) {
+return from(new Supplier() {
+  @Override
+  public ExecutorService get() {
+return staticExecutorService;
+  }
+});
+  }
+
+  public static TestExecutorService from(Supplier 
executorServiceSuppler) {
+return new FromSupplier(executorServiceSuppler);
+  }
+
+  /** A union of the {@link ExecutorService} and {@link TestRule} interfaces. 
*/
+  public interface TestExecutorService extends ExecutorService, TestRule {}
+
+  private static class FromSupplier extends ForwardingExecutorService
+  implements TestExecutorService {
+private final Supplier executorServiceSupplier;
+private ExecutorService delegate;
+
+private FromSupplier(Supplier executorServiceSupplier) {
+  this.executorServiceSupplier = executorServiceSupplier;
+}
+
+@Override
+public Statement apply(final Statement statement, Description arg1) {
+  return new Statement() {
+@Override
+public void evaluate() throws Throwable {
+  Throwable thrown = null;
+  delegate = executorServiceSupplier.get();
+  try {
+statement.evaluate();
+  } catch (Throwable t) {
+thrown = t;
+  }
+  shutdown();
+  if (!awaitTermination(5, TimeUnit.SECONDS)) {
+shutdownNow();
+IllegalStateException e =
+new IllegalStateException("Test executor failed to shutdown 
cleanly.");
+if (thrown != null) {
+  thrown.addSuppressed(e);
+} else {
+  thrown = e;
+}
+  }
+  if (thrown != null) {
+throw thrown;
+  }
+}
+  };
+}
+
+@Override
+protected ExecutorService delegate() {
+  return delegate;
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/f9e2be91/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java
--
diff --git 
a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java
 
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java
new file mode 100644
index 000..50c75b8
--- /dev/null
+++ 
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or 

[3/4] beam git commit: Move fn-execution org.apache.beam.harness.channel

[lcwik]

Move fn-execution org.apache.beam.harness.channel

'harness' suggests the SDK harness package, which is not correct, as
this is used by both runners and the SDK harness.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cbb3a730
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cbb3a730
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cbb3a730

Branch: refs/heads/master
Commit: cbb3a73086dccf16d6e2f521d99036aea17a0e1c
Parents: 5c74022
Author: Thomas Groh 
Authored: Thu Nov 9 17:06:23 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 16:16:41 2017 -0800

--
 .../beam/runners/fnexecution/ServerFactory.java |  2 +-
 .../runners/fnexecution/ServerFactoryTest.java  |  2 +-
 .../harness/channel/ManagedChannelFactory.java  | 82 
 .../harness/channel/SocketAddressFactory.java   | 64 ---
 .../beam/harness/channel/package-info.java  | 22 --
 .../sdk/fn/channel/ManagedChannelFactory.java   | 82 
 .../sdk/fn/channel/SocketAddressFactory.java| 64 +++
 .../beam/sdk/fn/channel/package-info.java   | 22 ++
 .../channel/ManagedChannelFactoryTest.java  | 71 -
 .../channel/SocketAddressFactoryTest.java   | 56 -
 .../fn/channel/ManagedChannelFactoryTest.java   | 71 +
 .../fn/channel/SocketAddressFactoryTest.java| 56 +
 .../org/apache/beam/fn/harness/FnHarness.java   |  2 +-
 13 files changed, 298 insertions(+), 298 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cbb3a730/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
--
diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
index 918672a..93c787d 100644
--- 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
@@ -28,7 +28,7 @@ import java.io.IOException;
 import java.net.InetAddress;
 import java.net.InetSocketAddress;
 import java.net.SocketAddress;
-import org.apache.beam.harness.channel.SocketAddressFactory;
+import org.apache.beam.sdk.fn.channel.SocketAddressFactory;
 import org.apache.beam.model.pipeline.v1.Endpoints;
 
 /**

http://git-wip-us.apache.org/repos/asf/beam/blob/cbb3a730/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
--
diff --git 
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
 
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
index b78e88a..e0d7bf9 100644
--- 
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
+++ 
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
@@ -38,7 +38,7 @@ import java.util.ArrayList;
 import java.util.Collection;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.LinkedBlockingQueue;
-import org.apache.beam.harness.channel.ManagedChannelFactory;
+import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
 import org.apache.beam.harness.test.Consumer;
 import org.apache.beam.harness.test.TestStreams;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi;

http://git-wip-us.apache.org/repos/asf/beam/blob/cbb3a730/sdks/java/fn-execution/src/main/java/org/apache/beam/harness/channel/ManagedChannelFactory.java
--
diff --git 
a/sdks/java/fn-execution/src/main/java/org/apache/beam/harness/channel/ManagedChannelFactory.java
 
b/sdks/java/fn-execution/src/main/java/org/apache/beam/harness/channel/ManagedChannelFactory.java
deleted file mode 100644
index 187cfdb..000
--- 
a/sdks/java/fn-execution/src/main/java/org/apache/beam/harness/channel/ManagedChannelFactory.java
+++ /dev/null
@@ -1,82 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3600

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 >  # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09
Commit message: "Fix test runtime class conflicts and get hbase tests passing."
 > git rev-list 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_Verify] $ /bin/bash -xe 
/tmp/jenkins7330466033884098725.sh
+ bash sdks/python/run_postcommit.sh
bash: sdks/python/run_postcommit.sh: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ajam...@ajamato2016.sea.corp.google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user valen...@google.com
Not sending mail to unregistered user git...@nakomis.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user 
ke...@kedin-macbookpro.roam.corp.google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user xumingmi...@gmail.com
ERROR: Unable to send to address: echauc...@gmail.com+jbono...@apache.org
javax.mail.internet.AddressException: Domain contains illegal character in 
string ``echauc...@gmail.com+jbono...@apache.org''
at 
javax.mail.internet.InternetAddress.checkAddress(InternetAddress.java:1269)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:1091)
at javax.mail.internet.InternetAddress.parse(InternetAddress.java:633)
at javax.mail.internet.InternetAddress.(InternetAddress.java:111)
at hudson.tasks.Mailer.stringToAddress(Mailer.java:206)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.toNormalizedAddress(MimeMessageBuilder.java:309)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:168)
at 
jenkins.plugins.mailer.tasks.MimeMessageBuilder.addRecipients(MimeMessageBuilder.java:160)
at hudson.tasks.MailSender.createEmptyMail(MailSender.java:389)
at hudson.tasks.MailSender.createFailureMail(MailSender.java:266)
at hudson.tasks.MailSender.createMail(MailSender.java:181)
at hudson.tasks.MailSender.run(MailSender.java:112)
at hudson.tasks.Mailer.perform(Mailer.java:170)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:682)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:627)
at hudson.model.Run.execute(Run.java:1762)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)

Build failed in Jenkins: beam_PerformanceTests_Python #589

[Apache Jenkins Server]

See 


Changes:

[altay] Update julia set example comments

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5c74022da1bb8f8a822cf3c545f96f2903d175a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5c74022da1bb8f8a822cf3c545f96f2903d175a4
Commit message: "This closes #4167"
 > git rev-list ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7173809017268764723.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6141693175475994484.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6713706107545700402.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3598

[Apache Jenkins Server]

See 


Changes:

[lcwik] Move fn-execution org.apache.beam.harness.channel

[lcwik] Move Common Fn Execution Concepts to fn-execution

--
[...truncated 767.74 KB...]
  e.exception.message)
ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263595#comment-16263595
 ] 

Luke Cwik commented on BEAM-3120:
-

The changes to the checkoutDir were reverted in 
https://github.com/apache/beam/commit/905facbb48e3e9fc4f4e48fbf65fd78474256185#diff-d72ab0cbaf787269f7822445cd1484b3

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3238) [SQL] Add builder to BeamRecordSqlType

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3238:
-

 Summary: [SQL] Add builder to BeamRecordSqlType
 Key: BEAM-3238
 URL: https://issues.apache.org/jira/browse/BEAM-3238
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin
Assignee: Anton Kedin


Currently it's hard to match field names with types when constructing a 
BeamRecordSqlType, like 
[here|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java#L64-L71]:

{code:java}
BeamRecordSqlType type = BeamRecordSqlType.create(
Arrays.asList("ts", "c_tinyint", "c_smallint",
"c_integer", "c_bigint", "c_float", "c_double", "c_decimal",
"c_tinyint_max", "c_smallint_max", "c_integer_max", "c_bigint_max"),
Arrays.asList(Types.DATE, Types.TINYINT, Types.SMALLINT,
Types.INTEGER, Types.BIGINT, Types.FLOAT, Types.DOUBLE, Types.DECIMAL,
Types.TINYINT, Types.SMALLINT, Types.INTEGER, Types.BIGINT)
);
{code}

It would be much more readable to have a builder, along these lines:
{code:java}
BeamRecordSqlType.builder()
  .withField("f_int", Types.INTEGER)
  .withStringField("f_str")
  .build();
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263637#comment-16263637
 ] 

Valentyn Tymofieiev edited comment on BEAM-3120 at 11/23/17 1:13 AM:
-

One of first line in Jenkins logs is 
 > git rev-parse --is-inside-work-tree # timeout=10
I suspect Git-SCM plugin is trying to find it's way into a Git repository. The 
workspace is dirty in the sense that it has two repositories, in workspace root 
and in /src, and the plugin picks the repository in workspace root, which does 
not have recent commits. I'll file an INFRA ticket to clean the workspace on 
the slaves per [~jasonkuster]'s suggestion.


was (Author: tvalentyn):
One of first line in Jenkins logs is 
 > git rev-parse --is-inside-work-tree # timeout=10
I suspect Git-SCM plugin is trying to find it's way into a Git repository. The 
workspace is dirty in the sense that it has two repositories, in workspace root 
and in /src, and the plugin picks the Workspace. I'll file an INFRA ticket to 
clean the workspace on the slaves per [~jasonkuster]'s suggestion.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263641#comment-16263641
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

Filed https://issues.apache.org/jira/browse/INFRA-15532.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[Chet Aldrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263472#comment-16263472
 ] 

Chet Aldrich commented on BEAM-3201:


[~echauchot] First of all, thanks for getting that all sorted for me.

[~nerdynick] 

{quote}That said, if you want to have dynamic index/type (meaning do not use 
ConnectionConfiguration.withIndex and ConnectionConfiguration.withType) and 
also dynamic id depending of the document itself, we should add 3 optional user 
defined functions so that the user can provide them. I guess it makes the 
withDocumentIdField(String fieldName) redundant. So we should not implement 
it.{quote}

According to what Etienne said here, it seems like if we want to go this route 
we may want to rethink the design for this, especially since I agree with him 
about not polluting the document payload. 

However, I'm not necessarily sold on why this is necessary in the first place. 
Could you ([~nerdynick]) elaborate more on why your use case requires 
dynamically changing the index and type that you're writing on a per-element 
basis? Why not just split up the elements and write to a separate index via a 
separate sink with a different `ConnectionConfiguration`? IMHO one write 
operation should write to only one index, since, for example, it'd be odd to be 
writing entries to two different DB tables depending on a given element instead 
of just splitting them up into separate PCollections and _then_ writing them 
out to the different tables with separate sinks. 

This opinion is only based on my current understanding of what you're trying to 
accomplish though. Feel free to enlighten me if an assumption I made about your 
use case is incorrect. 

Would appreciate input from both of you on whether this use case is needed, and 
if it is, whether we should rethink how we're approaching this so we don't 
pollute the document payload with metadata. 










> ElasticsearchIO should deal with documents id
> -
>
> Key: BEAM-3201
> URL: https://issues.apache.org/jira/browse/BEAM-3201
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Chet Aldrich
>
> Today the ESIO only inserts the payload of the ES documents. Elasticsearch 
> generates a document id for each record inserted. So each new insertion is 
> considered as a new document. Users want to be able to update documents using 
> the IO. So, for the write part of the IO, users should be able to provide a 
> document id so that they could update already stored documents. Providing an 
> id for the documents could also help the user on indempotency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4395

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1036

[Apache Jenkins Server]

See 


Changes:

[altay] Update julia set example comments

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5c74022da1bb8f8a822cf3c545f96f2903d175a4 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5c74022da1bb8f8a822cf3c545f96f2903d175a4
Commit message: "This closes #4167"
 > git rev-list ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4981037737698015440.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5485786280082303953.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7002903958544997460.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263594#comment-16263594
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

I suspect 
https://github.com/apache/beam/commit/6fe6bc7eb684cc7c953190e9a38330870ada0fdb#diff-d72ab0cbaf787269f7822445cd1484b3L41
 may be the reason.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3238) [SQL] Add builder to BeamRecordSqlType

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263746#comment-16263746
 ] 

ASF GitHub Bot commented on BEAM-3238:
--

GitHub user akedin opened a pull request:

https://github.com/apache/beam/pull/4168

[BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akedin/beam sql-record-type-builder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4168


commit 77769aa46dd9a972526f805f6c476c4039121b29
Author: Anton Kedin 
Date:   2017-11-23T00:04:05Z

[BEAM-3238][SQL] Move Coders to map in BeamRecordSqlType

Improve readability. All coders are supposed to be thread safe and
currently are backed by the static instances.

commit 96c2a67f38c3da501f37797a8f75c23d5baf2ca8
Author: Anton Kedin 
Date:   2017-11-23T03:17:21Z

[BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder

To improve readability of creating BeamRecordSqlTypes.




> [SQL] Add builder to BeamRecordSqlType
> --
>
> Key: BEAM-3238
> URL: https://issues.apache.org/jira/browse/BEAM-3238
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> Currently it's hard to match field names with types when constructing a 
> BeamRecordSqlType, like 
> [here|https://github.com/apache/beam/blob/39e66e953b0f8e16435acb038cad364acf2b3a57/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java#L64-L71]:
> {code:java}
> BeamRecordSqlType type = BeamRecordSqlType.create(
> Arrays.asList("ts", "c_tinyint", "c_smallint",
> "c_integer", "c_bigint", "c_float", "c_double", "c_decimal",
> "c_tinyint_max", "c_smallint_max", "c_integer_max", "c_bigint_max"),
> Arrays.asList(Types.DATE, Types.TINYINT, Types.SMALLINT,
> Types.INTEGER, Types.BIGINT, Types.FLOAT, Types.DOUBLE, Types.DECIMAL,
> Types.TINYINT, Types.SMALLINT, Types.INTEGER, Types.BIGINT)
> );
> {code}
> It would be much more readable to have a builder, along these lines:
> {code:java}
> BeamRecordSqlType.builder()
>   .withField("f_int", Types.INTEGER)
>   .withStringField("f_str")
>   .build();
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4168: [BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder

[akedin]

GitHub user akedin opened a pull request:

https://github.com/apache/beam/pull/4168

[BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akedin/beam sql-record-type-builder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4168


commit 77769aa46dd9a972526f805f6c476c4039121b29
Author: Anton Kedin 
Date:   2017-11-23T00:04:05Z

[BEAM-3238][SQL] Move Coders to map in BeamRecordSqlType

Improve readability. All coders are supposed to be thread safe and
currently are backed by the static instances.

commit 96c2a67f38c3da501f37797a8f75c23d5baf2ca8
Author: Anton Kedin 
Date:   2017-11-23T03:17:21Z

[BEAM-3238][SQL] Add BeamRecordSqlTypeBuilder

To improve readability of creating BeamRecordSqlTypes.




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3597

[Apache Jenkins Server]

See 


Changes:

[altay] Update julia set example comments

--
[...truncated 779.45 KB...]
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok

Build failed in Jenkins: beam_PostCommit_Python_Verify #3599

[Apache Jenkins Server]

See 


Changes:

[lcwik] Fix test runtime class conflicts and get hbase tests passing.

--
[...truncated 815.34 KB...]
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying 

[2/4] beam git commit: Move Common Fn Execution Concepts to fn-execution

[lcwik]

Move Common Fn Execution Concepts to fn-execution

Move Stream control packages to fn-execution. Update java package names.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f9e2be91
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f9e2be91
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f9e2be91

Branch: refs/heads/master
Commit: f9e2be91836fd7fb25ed772a35d9626a7c5a2cc6
Parents: cbb3a73
Author: Thomas Groh 
Authored: Thu Nov 9 18:31:26 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 16:16:41 2017 -0800

--
 runners/java-fn-execution/pom.xml   |  29 ---
 .../beam/runners/fnexecution/ServerFactory.java |   2 +-
 .../runners/fnexecution/ServerFactoryTest.java  |   6 +-
 sdks/java/fn-execution/build.gradle |  10 +-
 sdks/java/fn-execution/pom.xml  |  49 ++--
 .../beam/sdk/fn/stream/AdvancingPhaser.java |  36 +++
 .../sdk/fn/stream/BufferingStreamObserver.java  | 171 +
 .../apache/beam/sdk/fn/stream/DataStreams.java  | 249 +++
 .../sdk/fn/stream/DirectStreamObserver.java |  71 ++
 .../ForwardingClientResponseObserver.java   |  68 +
 .../apache/beam/sdk/fn/stream/package-info.java |  22 ++
 .../org/apache/beam/harness/test/Consumer.java  |  26 --
 .../org/apache/beam/harness/test/Supplier.java  |  26 --
 .../apache/beam/harness/test/TestExecutors.java |  93 ---
 .../beam/harness/test/TestExecutorsTest.java| 175 -
 .../apache/beam/harness/test/TestStreams.java   | 185 --
 .../beam/harness/test/TestStreamsTest.java  | 109 
 .../beam/sdk/fn/stream/AdvancingPhaserTest.java |  53 
 .../fn/stream/BufferingStreamObserverTest.java  | 155 
 .../beam/sdk/fn/stream/DataStreamsTest.java | 167 +
 .../sdk/fn/stream/DirectStreamObserverTest.java | 145 +++
 .../ForwardingClientResponseObserverTest.java   |  60 +
 .../org/apache/beam/sdk/fn/test/Consumer.java   |  26 ++
 .../org/apache/beam/sdk/fn/test/Supplier.java   |  26 ++
 .../apache/beam/sdk/fn/test/TestExecutors.java  |  93 +++
 .../beam/sdk/fn/test/TestExecutorsTest.java | 175 +
 .../apache/beam/sdk/fn/test/TestStreams.java| 185 ++
 .../beam/sdk/fn/test/TestStreamsTest.java   | 109 
 .../org/apache/beam/fn/harness/FnHarness.java   |   2 +-
 .../beam/fn/harness/state/BagUserState.java |   2 +-
 .../beam/fn/harness/stream/AdvancingPhaser.java |  36 ---
 .../harness/stream/BufferingStreamObserver.java | 166 -
 .../beam/fn/harness/stream/DataStreams.java | 229 -
 .../fn/harness/stream/DirectStreamObserver.java |  71 --
 .../ForwardingClientResponseObserver.java   |  63 -
 .../harness/stream/StreamObserverFactory.java   |  20 +-
 .../fn/harness/BeamFnDataReadRunnerTest.java|   4 +-
 .../apache/beam/fn/harness/FnHarnessTest.java   |   4 +-
 .../control/BeamFnControlClientTest.java|   2 +-
 .../fn/harness/control/RegisterHandlerTest.java |   4 +-
 ...BeamFnDataBufferingOutboundObserverTest.java |   2 +-
 .../harness/data/BeamFnDataGrpcClientTest.java  |   4 +-
 .../data/BeamFnDataGrpcMultiplexerTest.java |   2 +-
 .../logging/BeamFnLoggingClientTest.java|   2 +-
 .../state/BeamFnStateGrpcClientCacheTest.java   |   2 +-
 .../fn/harness/stream/AdvancingPhaserTest.java  |  48 
 .../stream/BufferingStreamObserverTest.java | 146 ---
 .../beam/fn/harness/stream/DataStreamsTest.java | 167 -
 .../stream/DirectStreamObserverTest.java| 139 ---
 .../ForwardingClientResponseObserverTest.java   |  60 -
 .../stream/StreamObserverFactoryTest.java   |   3 +
 51 files changed, 1865 insertions(+), 1834 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f9e2be91/runners/java-fn-execution/pom.xml
--
diff --git a/runners/java-fn-execution/pom.xml 
b/runners/java-fn-execution/pom.xml
index 6ff08b7..3ebcfd0 100644
--- a/runners/java-fn-execution/pom.xml
+++ b/runners/java-fn-execution/pom.xml
@@ -32,35 +32,6 @@
 
   jar
 
-  
-
-  
-org.apache.maven.plugins
-maven-enforcer-plugin
-
-  
-enforce-banned-dependencies
-
-  enforce
-
-
-  
-
-  
-com.google.guava:guava-jdk5
-com.google.protobuf:protobuf-lite
-org.apache.beam:beam-sdks-java-core
-  
-
-  
-  true
-
-  
-
-  
-
-  
-
   
 
   org.apache.beam

[jira] [Commented] (BEAM-2899) Universal Local Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263604#comment-16263604
 ] 

ASF GitHub Bot commented on BEAM-2899:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4111


> Universal Local Runner
> --
>
> Key: BEAM-2899
> URL: https://issues.apache.org/jira/browse/BEAM-2899
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>  Labels: portability
>
> To make the portability effort tractable, we should implement a Universal 
> Local Runner (ULR) in Java that runs in a single server process plus docker 
> containers for the SDK harness containers. It would serve multiple purposes:
>   (1) A reference implementation for other runners. Ideally, any new feature 
> should be implemented in the ULR first.
>   (2) A fully-featured test runner for SDKs who participate in the 
> portability framework. It thus complements the direct runners.
>   (3) A test runner for user code that depends on or customizes the runtime 
> environment. For example, a DoFn that shells out has a dependency that may be 
> satisfied on the user's desktop (and thus works fine on the direct runner), 
> but perhaps not by the container harness image. The ULR allows for an easy 
> way to find out.
> The Java direct runner presumably has lots of pieces that can be reused.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4111: [BEAM-2899] Migrate Stream Control to the fn-execut...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4111


---

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263624#comment-16263624
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

Thanks; that explains why the Jenkins workspace has git sources up to  
https://github.com/apache/beam/commit/6fe6bc7 and also a src/ folder that has 
latest changes. I suspect Jenkins may still pick up sources from the workspace  
when running bash sdks/python/run_postcommit.sh.


> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4392

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3226) Gremlin IO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3226:
---
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> Gremlin IO
> --
>
> Key: BEAM-3226
> URL: https://issues.apache.org/jira/browse/BEAM-3226
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-java-extensions
>Reporter: Utkarsh Sopan
>Assignee: Kenneth Knowles
>
> IO transform for graph databases, better to provide a gremlin io.
> http://tinkerpop.apache.org/docs/current/reference/#_features



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Spark #1034

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision a40bdfd756e5b187a6c7fea86c253efd0feb1701 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a40bdfd756e5b187a6c7fea86c253efd0feb1701
Commit message: "This closes #4158"
 > git rev-list a40bdfd756e5b187a6c7fea86c253efd0feb1701 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6064798126263838153.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5541191051582172276.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2167531441943308436.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 

[jira] [Commented] (BEAM-3201) ElasticsearchIO should deal with documents id

[Nicholas Verbeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263519#comment-16263519
 ] 

Nicholas Verbeck commented on BEAM-3201:


[~chetaldrich] I'm not against the user functions. I just feel to really 
support them efficiently the ESIO.Write() signature would need to change from 
String to something else; Map, Object, JSONObject, etc. In fact I talked about 
it when I tried to start the discussions on BEAM-3222 within the dev mailing 
list. 

The use-case I'm trying to solve for is not a unique one or a new of one. In 
most cases, including my own currently, it involves timeseries data. Where 
you'd bucket the data into separate indexes by day, hour, etc. It'd be 
impractical to launch separate jobs or define an unlimited list of partitions 
for each time separation. Especially when streaming data from Kafka. Data shows 
up late and other issues would make it very difficult if you couldn't change 
the index/type dynamically as the data flows by. ES already supports this 
action/use-case with the Bulk API. As well as further enhances the ability to 
do this with index templates. 

> ElasticsearchIO should deal with documents id
> -
>
> Key: BEAM-3201
> URL: https://issues.apache.org/jira/browse/BEAM-3201
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Chet Aldrich
>
> Today the ESIO only inserts the payload of the ES documents. Elasticsearch 
> generates a document id for each record inserted. So each new insertion is 
> considered as a new document. Users want to be able to update documents using 
> the IO. So, for the write part of the IO, users should be able to provide a 
> document id so that they could update already stored documents. Providing an 
> id for the documents could also help the user on indempotency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263637#comment-16263637
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

One of first line in Jenkins logs is 
 > git rev-parse --is-inside-work-tree # timeout=10
I suspect Git-SCM plugin is trying to find it's way into a Git repository. The 
workspace is dirty in the sense that it has two repositories, in workspace root 
and in /src, and the plugin picks the Workspace. I'll file an INFRA ticket to 
clean the workspace on the slaves per [~jasonkuster]'s suggestion.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3239) Enable debug server for python sdk workers

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-3239:
--

 Summary: Enable debug server for python sdk workers
 Key: BEAM-3239
 URL: https://issues.apache.org/jira/browse/BEAM-3239
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Ankur Goenka
Assignee: Ankur Goenka
Priority: Minor


Enable status server to dump threads when http get call is made to the server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Spark #1037

[Apache Jenkins Server]

See 


Changes:

[lcwik] Move fn-execution org.apache.beam.harness.channel

[lcwik] Move Common Fn Execution Concepts to fn-execution

[lcwik] Fix test runtime class conflicts and get hbase tests passing.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init  
 > # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09
Commit message: "Fix test runtime class conflicts and get hbase tests passing."
 > git rev-list 5c74022da1bb8f8a822cf3c545f96f2903d175a4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5453307531073204474.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8380163989257778130.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6571200454855805284.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

Build failed in Jenkins: beam_PerformanceTests_Python #590

[Apache Jenkins Server]

See 


Changes:

[lcwik] Move fn-execution org.apache.beam.harness.channel

[lcwik] Move Common Fn Execution Concepts to fn-execution

[lcwik] Fix test runtime class conflicts and get hbase tests passing.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init  
 > # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 558aa2fc3efda58a2cd398b20a16cf9b6ca3ef09
Commit message: "Fix test runtime class conflicts and get hbase tests passing."
 > git rev-list 5c74022da1bb8f8a822cf3c545f96f2903d175a4 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3554630130424240449.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3117166450054378151.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4479566041830754632.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

[jira] [Assigned] (BEAM-2543) HBaseIOTest fails tests if run from a directory with spaces

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-2543:
--

Assignee: (was: Ismaël Mejía)

> HBaseIOTest fails tests if run from a directory with spaces
> ---
>
> Key: BEAM-2543
> URL: https://issues.apache.org/jira/browse/BEAM-2543
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>
> HBaseIOTest failed Jenkins Java cross-JDK tests since June 22 as can tell 
> from Jenkins history. Both JDK7, OpenJDK7&8 are affected.
> Errors:
> {code}
> 2017-06-29\T\18:13:56.187 [ERROR] org.apache.beam.sdk.io.hbase.HBaseIOTest  
> Time elapsed: 0 s  <<< ERROR!
> java.io.IOException: Shutting down
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:235)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:97)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1002)
>   at 
> org.apache.beam.sdk.io.hbase.HBaseIOTest.beforeClass(HBaseIOTest.java:102)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed construction of Master: class 
> org.apache.hadoop.hbase.master.HMasterIllegal character in path at index 89: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.7 
> (latest)/label/beam/sdks/java/io/hbase/target/test-data/b11a0828-4628-4fe9-885d-073fb641ddc9
>   at 
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:217)
>   ... 23 more
> Caused by: java.lang.IllegalArgumentException: Illegal character in path at 
> index 89: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.7 
> (latest)/label/beam/sdks/java/io/hbase/target/test-data/b11a0828-4628-4fe9-885d-073fb641ddc9
>   at java.net.URI.create(URI.java:859)
>   at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:175)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
>   at org.apache.hadoop.hbase.fs.HFileSystem.(HFileSystem.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:613)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:564)
>   at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:412)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
>   ... 26 more
> Caused by: 

[jira] [Updated] (BEAM-2296) HBase IO should use AutoValue

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-2296:
---
Labels: newbie starter  (was: )

> HBase IO should use AutoValue
> -
>
> Key: BEAM-2296
> URL: https://issues.apache.org/jira/browse/BEAM-2296
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Borisa Zivkovic
>Assignee: Borisa Zivkovic
>  Labels: newbie, starter
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3185) Build blocks on parsing long as int from github status json

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3185:
--

Assignee: (was: Ismaël Mejía)

> Build blocks on parsing long as int from github status json
> ---
>
> Key: BEAM-3185
> URL: https://issues.apache.org/jira/browse/BEAM-3185
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: holdenk
>Priority: Blocker
>
> (e.g. see 
> https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/818/console )
> `Caused by: com.fasterxml.jackson.databind.JsonMappingException: Numeric 
> value (4313677368) out of range of int`
> Assuming IDs are monotonically increasing this might impact all new PRs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2296) HBase IO should use AutoValue

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262541#comment-16262541
 ] 

Ismaël Mejía commented on BEAM-2296:


Hi, I would like to know if you still plan to do this or if we can let it open 
so anyone else can take this task

> HBase IO should use AutoValue
> -
>
> Key: BEAM-2296
> URL: https://issues.apache.org/jira/browse/BEAM-2296
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Borisa Zivkovic
>Assignee: Borisa Zivkovic
>  Labels: newbie, starter
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3236) Add OrientDbIO

[JIRA]

Jean-Baptiste Onofré created BEAM-3236:
--

 Summary: Add OrientDbIO
 Key: BEAM-3236
 URL: https://issues.apache.org/jira/browse/BEAM-3236
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3215) Add a performance test for HBaseIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262558#comment-16262558
 ] 

Ismaël Mejía commented on BEAM-3215:


Hi thanks for the explanation [~chamikara] I don't plan to actively work on 
this, I think is a great idea to have more people contributing in the different 
IOs so I stay away. Anyway don't hesitate to contact me or ping me if anything 
is needed.

> Add a performance test for HBaseIO
> --
>
> Key: BEAM-3215
> URL: https://issues.apache.org/jira/browse/BEAM-3215
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-extensions
>Reporter: Chamikara Jayalath
>Assignee: Kamil Szewczyk
>
> We should add a large scale performance test for HBaseIO. We could use 
> PerfKitBenchmarker based performance testing framework [1] to manage a 
> Kubernetes based muti-node HBase cluster and to publish benchmark results.
> Example docker image to use: https://hub.docker.com/r/dajobe/hbase/
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2543) HBaseIOTest fails tests if run from a directory with spaces

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262568#comment-16262568
 ] 

Ismaël Mejía commented on BEAM-2543:


I am unassigning myself from this one since there is nothing I can actively do 
to fix it because I don't have permissions to change the symbolic links in 
jenkins, and upstream HBase does not seem to consider fixing directories with 
spaces as a priority.

> HBaseIOTest fails tests if run from a directory with spaces
> ---
>
> Key: BEAM-2543
> URL: https://issues.apache.org/jira/browse/BEAM-2543
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Ismaël Mejía
>
> HBaseIOTest failed Jenkins Java cross-JDK tests since June 22 as can tell 
> from Jenkins history. Both JDK7, OpenJDK7&8 are affected.
> Errors:
> {code}
> 2017-06-29\T\18:13:56.187 [ERROR] org.apache.beam.sdk.io.hbase.HBaseIOTest  
> Time elapsed: 0 s  <<< ERROR!
> java.io.IOException: Shutting down
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:235)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:97)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1002)
>   at 
> org.apache.beam.sdk.io.hbase.HBaseIOTest.beforeClass(HBaseIOTest.java:102)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed construction of Master: class 
> org.apache.hadoop.hbase.master.HMasterIllegal character in path at index 89: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.7 
> (latest)/label/beam/sdks/java/io/hbase/target/test-data/b11a0828-4628-4fe9-885d-073fb641ddc9
>   at 
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:220)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
>   at 
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:217)
>   ... 23 more
> Caused by: java.lang.IllegalArgumentException: Illegal character in path at 
> index 89: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/JDK
>  1.7 
> (latest)/label/beam/sdks/java/io/hbase/target/test-data/b11a0828-4628-4fe9-885d-073fb641ddc9
>   at java.net.URI.create(URI.java:859)
>   at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:175)
>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
>   at org.apache.hadoop.hbase.fs.HFileSystem.(HFileSystem.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.initializeFileSystem(HRegionServer.java:613)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:564)
>   at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:412)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> 

[GitHub] beam pull request #4164: Update DataflowRunner fnapi environment version

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/4164

Update DataflowRunner fnapi environment version

R: @tvalentyn 
cc: @mdvorsky

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4164.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4164


commit 62768d35209f16dd93caf5e2ec2c833a09f32c3b
Author: Ahmet Altay 
Date:   2017-11-22T15:49:52Z

Update DataflowRunner fnapi environment version




---

[jira] [Assigned] (BEAM-3226) Gremlin IO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3226:
--

Assignee: (was: Kenneth Knowles)

> Gremlin IO
> --
>
> Key: BEAM-3226
> URL: https://issues.apache.org/jira/browse/BEAM-3226
> Project: Beam
>  Issue Type: Wish
>  Components: sdk-java-extensions
>Reporter: Utkarsh Sopan
>
> IO transform for graph databases, better to provide a gremlin io.
> http://tinkerpop.apache.org/docs/current/reference/#_features



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-1783) Add Integration Tests for HBaseIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-1783.

   Resolution: Fixed
Fix Version/s: Not applicable

> Add Integration Tests for HBaseIO
> -
>
> Key: BEAM-1783
> URL: https://issues.apache.org/jira/browse/BEAM-1783
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-extensions, testing
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3595

[Apache Jenkins Server]

See 


--
[...truncated 817.25 KB...]
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker

[jira] [Commented] (BEAM-3180) [Nexmark][SQL] Refactor NexmarkLauncher

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263006#comment-16263006
 ] 

ASF GitHub Bot commented on BEAM-3180:
--

GitHub user akedin opened a pull request:

https://github.com/apache/beam/pull/4165

[BEAM-3180][NexmarkLauncher] Extract EventSources

Extracting event sources out from NexmarkLauncher.

Refactoring to make the launcher smaller.

This is the first commit, trying to split [the previous large 
PR](https://github.com/apache/beam/pull/4110) into smaller chunks. Plus this is 
rebased on top of master.


Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akedin/beam nexmark-generator-refactor-rebase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4165


commit 5cc1ca934cc672b3d5e97ee405ab586e08b32c93
Author: Anton Kedin 
Date:   2017-11-07T21:44:58Z

[BEAM-3147][NexmarkLauncher] Extract EventSourceFactory and Event sources 
out from NexmarkLauncher

Refactoring to make the launcher smaller.




> [Nexmark][SQL] Refactor NexmarkLauncher
> ---
>
> Key: BEAM-3180
> URL: https://issues.apache.org/jira/browse/BEAM-3180
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>
> NexmarkLauncher is a huge class which handles everything about running the 
> queries, from generation of entities, managing sources and sinks, to 
> monitoring the perf. And it has zero coverage.
> It needs to be split into few components, at least need to move event 
> generation, sources and sinks creation, and monitoring into their own 
> components.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4165: [BEAM-3180][NexmarkLauncher] Extract EventSources

[akedin]

GitHub user akedin opened a pull request:

https://github.com/apache/beam/pull/4165

[BEAM-3180][NexmarkLauncher] Extract EventSources

Extracting event sources out from NexmarkLauncher.

Refactoring to make the launcher smaller.

This is the first commit, trying to split [the previous large 
PR](https://github.com/apache/beam/pull/4110) into smaller chunks. Plus this is 
rebased on top of master.


Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akedin/beam nexmark-generator-refactor-rebase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4165


commit 5cc1ca934cc672b3d5e97ee405ab586e08b32c93
Author: Anton Kedin 
Date:   2017-11-07T21:44:58Z

[BEAM-3147][NexmarkLauncher] Extract EventSourceFactory and Event sources 
out from NexmarkLauncher

Refactoring to make the launcher smaller.




---

[GitHub] beam pull request #4164: Update DataflowRunner fnapi environment version

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/4164


---

[2/2] beam git commit: Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle build.

[lcwik]

Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle build.

This closes #4163


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ea33f2a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ea33f2a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ea33f2a4

Branch: refs/heads/master
Commit: ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b
Parents: a40bdfd 982605d
Author: Luke Cwik 
Authored: Wed Nov 22 08:54:59 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 08:54:59 2017 -0800

--
 sdks/java/io/hadoop/jdk1.8-tests/build.gradle | 35 +-
 1 file changed, 27 insertions(+), 8 deletions(-)
--

[1/2] beam git commit: Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle build.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master a40bdfd75 -> ea33f2a4a


Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle build.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/982605d9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/982605d9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/982605d9

Branch: refs/heads/master
Commit: 982605d922217e606f0cd5e27bf3b6a88a2f594e
Parents: a40bdfd
Author: Luke Cwik 
Authored: Tue Nov 21 18:02:55 2017 -0800
Committer: Luke Cwik 
Committed: Wed Nov 22 08:54:39 2017 -0800

--
 sdks/java/io/hadoop/jdk1.8-tests/build.gradle | 35 +-
 1 file changed, 27 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/982605d9/sdks/java/io/hadoop/jdk1.8-tests/build.gradle
--
diff --git a/sdks/java/io/hadoop/jdk1.8-tests/build.gradle 
b/sdks/java/io/hadoop/jdk1.8-tests/build.gradle
index 25d0d85..14ea897 100644
--- a/sdks/java/io/hadoop/jdk1.8-tests/build.gradle
+++ b/sdks/java/io/hadoop/jdk1.8-tests/build.gradle
@@ -30,16 +30,22 @@ description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop 
:: jdk1.8-tests"
  */
 
evaluationDependsOn(":beam-sdks-parent:beam-sdks-java-parent:beam-sdks-java-io-parent:beam-sdks-java-io-common")
 
-test {
-  // TODO: Get tests to run. Known issues:
-  //  * hive-exec bundles w/o repackaging Guava (HIVE-13690)
-  ignoreFailures true
-}
-
 def log4j_version = "2.6.2"
 def elastic_search_version = "5.0.0"
+// Migrate to using a version of the driver compatible with Guava 20
+def cassandra_driver = "3.2.0"
+
+// Ban dependencies from the test runtime classpath
+configurations.testRuntimeClasspath {
+  // Ban hive-exec and mesos since they bundle protobuf without repackaging
+  exclude group: "org.apache.hive", module: "hive-exec"
+  exclude group: "org.apache.mesos", module: "mesos"
+  // Prevent a StackOverflow because of wiring LOG4J -> SLF4J -> LOG4J
+  exclude group: "org.slf4j", module: "log4j-over-slf4j"
+}
 
 dependencies {
+  shadow library.java.protobuf_java
   shadow project(path: 
":beam-sdks-parent:beam-sdks-java-parent:beam-sdks-java-core", configuration: 
"shadow")
   shadow project(path: 
":beam-sdks-parent:beam-sdks-java-parent:beam-sdks-java-io-parent:beam-sdks-java-io-hadoop-parent:beam-sdks-java-io-hadoop-input-format",
 configuration: "shadow")
   shadow library.java.slf4j_api
@@ -55,9 +61,9 @@ dependencies {
   testCompile "org.elasticsearch.client:transport:$elastic_search_version"
   testCompile "io.netty:netty-transport-native-epoll:4.1.0.CR3"
   testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
-  testCompile "com.datastax.cassandra:cassandra-driver-mapping:3.1.1"
+  testCompile 
"com.datastax.cassandra:cassandra-driver-mapping:$cassandra_driver"
   testCompile "org.apache.cassandra:cassandra-all:3.9"
-  testCompile "com.datastax.cassandra:cassandra-driver-core:3.1.1"
+  testCompile "com.datastax.cassandra:cassandra-driver-core:$cassandra_driver"
   testCompile "org.apache.logging.log4j:log4j-core:$log4j_version"
   testCompile library.java.junit
   testCompile library.java.mockito_core
@@ -69,3 +75,16 @@ task packageTests(type: Jar) {
 }
 
 artifacts.archives packageTests
+
+// The cassandra.yaml file currently assumes "target/..." exists.
+// TODO: Update cassandra.yaml to inject new properties representing
+// the root path. Also migrate cassandra.yaml to use any open ports
+// instead of a static port.
+task createTargetDirectoryForCassandra() {
+  doLast {
+if (!project.file("target").exists()) {
+  project.file("target").mkdirs();
+}
+  }
+}
+test.dependsOn createTargetDirectoryForCassandra

[GitHub] beam pull request #4163: Update sdks/java/io/hadoop/jdk1.8-tests to run and ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4163


---

[jira] [Commented] (BEAM-3185) Build blocks on parsing long as int from github status json

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262914#comment-16262914
 ] 

Ismaël Mejía commented on BEAM-3185:


Reassigning this to its previous 'asigneé' sorry I took it by mistake when 
reading about the issue.

> Build blocks on parsing long as int from github status json
> ---
>
> Key: BEAM-3185
> URL: https://issues.apache.org/jira/browse/BEAM-3185
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: holdenk
>Assignee: Jason Kuster
>Priority: Blocker
>
> (e.g. see 
> https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/818/console )
> `Caused by: com.fasterxml.jackson.databind.JsonMappingException: Numeric 
> value (4313677368) out of range of int`
> Assuming IDs are monotonically increasing this might impact all new PRs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3185) Build blocks on parsing long as int from github status json

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3185:
--

Assignee: Jason Kuster

> Build blocks on parsing long as int from github status json
> ---
>
> Key: BEAM-3185
> URL: https://issues.apache.org/jira/browse/BEAM-3185
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: holdenk
>Assignee: Jason Kuster
>Priority: Blocker
>
> (e.g. see 
> https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/818/console )
> `Caused by: com.fasterxml.jackson.databind.JsonMappingException: Numeric 
> value (4313677368) out of range of int`
> Assuming IDs are monotonically increasing this might impact all new PRs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263025#comment-16263025
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

I am no longer sure which code Jenkins Python postcommit tests suite is using 
at all.
Let's look at https://builds.apache.org/job/beam_PostCommit_Python_Verify/3595/

Jenkins says: 
  Revision: a40bdfd756e5b187a6c7fea86c253efd0feb1701
  origin/master

We have a failing integration test, test_wordcount_fnapi_it 
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
Job logs: 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-11-22_07_29_57-11055969376023337206?project=apache-beam-testing

Job logs show that we are using Container {Name:sdk 
Image:dataflow.gcr.io/v1beta3/python-fnapi:beam-2.1.0-20170621 ... }

Looking at git show 
a40bdfd756e5b187a6c7fea86c253efd0feb1701:./apache_beam/runners/dataflow/internal/dependency.py
 | grep BEAM_FNAPI_CONTAINER_VERSION

we can see BEAM_FNAPI_CONTAINER_VERSION = 'beam-2.3.0-20171121' . It's not 
clear which revision Jenkins is using...

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4393

[Apache Jenkins Server]

See 

[GitHub] beam pull request #4166: Fix test runtime class conflicts and get hbase test...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4166

Fix test runtime class conflicts and get hbase tests passing.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam gradle

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4166.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4166






---

[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263025#comment-16263025
 ] 

Valentyn Tymofieiev edited comment on BEAM-3120 at 11/22/17 6:57 PM:
-

I am no longer sure which code Jenkins Python postcommit tests suite is using 
at all.
Let's look at https://builds.apache.org/job/beam_PostCommit_Python_Verify/3595/ 
that was triggered for the master branch.

Jenkins says: 
  Revision: a40bdfd756e5b187a6c7fea86c253efd0feb1701
  origin/master

We have a failing integration test, test_wordcount_fnapi_it 
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
Job logs: 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-11-22_07_29_57-11055969376023337206?project=apache-beam-testing

Job logs show that we are using Container {Name:sdk 
Image:dataflow.gcr.io/v1beta3/python-fnapi:beam-2.1.0-20170621 ... }

Looking at git show 
a40bdfd756e5b187a6c7fea86c253efd0feb1701:./apache_beam/runners/dataflow/internal/dependency.py
 | grep BEAM_FNAPI_CONTAINER_VERSION

we can see BEAM_FNAPI_CONTAINER_VERSION = 'beam-2.3.0-20171121' . It's not 
clear which revision Jenkins is using...


was (Author: tvalentyn):
I am no longer sure which code Jenkins Python postcommit tests suite is using 
at all.
Let's look at https://builds.apache.org/job/beam_PostCommit_Python_Verify/3595/

Jenkins says: 
  Revision: a40bdfd756e5b187a6c7fea86c253efd0feb1701
  origin/master

We have a failing integration test, test_wordcount_fnapi_it 
(apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR
Job logs: 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2017-11-22_07_29_57-11055969376023337206?project=apache-beam-testing

Job logs show that we are using Container {Name:sdk 
Image:dataflow.gcr.io/v1beta3/python-fnapi:beam-2.1.0-20170621 ... }

Looking at git show 
a40bdfd756e5b187a6c7fea86c253efd0feb1701:./apache_beam/runners/dataflow/internal/dependency.py
 | grep BEAM_FNAPI_CONTAINER_VERSION

we can see BEAM_FNAPI_CONTAINER_VERSION = 'beam-2.3.0-20171121' . It's not 
clear which revision Jenkins is using...

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3237) Logs from python custom commands are not visible at installation time

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-3237:
-

 Summary: Logs from python custom commands are not visible at 
installation time
 Key: BEAM-3237
 URL: https://issues.apache.org/jira/browse/BEAM-3237
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay
Priority: Minor


Outputs from the custom commands in the julia set example does not show in the 
DataflowRunner logs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Spark #1035

[Apache Jenkins Server]

See 


Changes:

[lcwik] Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b
Commit message: "Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for 
Gradle build."
 > git rev-list a40bdfd756e5b187a6c7fea86c253efd0feb1701 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins778827815480955973.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7474525441702351176.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4426303138037009619.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3596

[Apache Jenkins Server]

See 


Changes:

[lcwik] Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle

--
[...truncated 811.11 KB...]
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/worker
copying 

Build failed in Jenkins: beam_PerformanceTests_Python #588

[Apache Jenkins Server]

See 


Changes:

[lcwik] Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for Gradle

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ea33f2a4ac4929072fe99eb9b99e4adbb38c8f9b
Commit message: "Update sdks/java/io/hadoop/jdk1.8-tests to run and pass for 
Gradle build."
 > git rev-list a40bdfd756e5b187a6c7fea86c253efd0feb1701 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8083554022083225739.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7151549115510282096.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8427381907665795927.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263146#comment-16263146
 ] 

Ahmet Altay commented on BEAM-3120:
---

The value does not match release branch either: 
https://github.com/apache/beam/blob/release-2.2.0/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L85

Is it possible that Jenkins is running from a not-updated git repository?

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263154#comment-16263154
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

I'll try to get the staged SDK, to see if that will help. Other than that, not 
sure what other information we can get without access to Jenkins machines.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: This closes #4167

[altay]

This closes #4167


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5c74022d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5c74022d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5c74022d

Branch: refs/heads/master
Commit: 5c74022da1bb8f8a822cf3c545f96f2903d175a4
Parents: ea33f2a 7fe7732
Author: Ahmet Altay 
Authored: Wed Nov 22 13:07:29 2017 -0800
Committer: Ahmet Altay 
Committed: Wed Nov 22 13:07:29 2017 -0800

--
 .../examples/complete/juliaset/juliaset_main.py   |  2 +-
 .../apache_beam/examples/complete/juliaset/setup.py   | 10 +-
 2 files changed, 10 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Update julia set example comments

[altay]

Repository: beam
Updated Branches:
  refs/heads/master ea33f2a4a -> 5c74022da


Update julia set example comments


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7fe7732b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7fe7732b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7fe7732b

Branch: refs/heads/master
Commit: 7fe7732bbf7721231ef7ec072307e533f006aef3
Parents: ea33f2a
Author: Ahmet Altay 
Authored: Wed Nov 22 11:43:05 2017 -0800
Committer: Ahmet Altay 
Committed: Wed Nov 22 11:49:50 2017 -0800

--
 .../examples/complete/juliaset/juliaset_main.py   |  2 +-
 .../apache_beam/examples/complete/juliaset/setup.py   | 10 +-
 2 files changed, 10 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7fe7732b/sdks/python/apache_beam/examples/complete/juliaset/juliaset_main.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/juliaset/juliaset_main.py 
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset_main.py
index 1d521be..7289b04 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/juliaset_main.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/juliaset_main.py
@@ -43,7 +43,7 @@ python juliaset_main.py \
   --staging_location gs://YOUR-BUCKET/juliaset/staging \
   --temp_location gs://YOUR-BUCKET/juliaset/temp \
   --coordinate_output gs://YOUR-BUCKET/juliaset/out \
-  --grid_size 20 \
+  --grid_size 20
 
 """
 

http://git-wip-us.apache.org/repos/asf/beam/blob/7fe7732b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
--
diff --git a/sdks/python/apache_beam/examples/complete/juliaset/setup.py 
b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
index cbf5f3d..2fcde7d 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/setup.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
@@ -51,7 +51,7 @@ class build(_build):  # pylint: disable=invalid-name
 # two commands will have to be added:
 #
 # ['apt-get', 'update'],
-# ['apt-get', '--assume-yes', install', 'libjpeg62'],
+# ['apt-get', '--assume-yes', 'install', 'libjpeg62'],
 #
 # First, note that there is no need to use the sudo command because the setup
 # script runs with appropriate access.
@@ -61,6 +61,14 @@ class build(_build):  # pylint: disable=invalid-name
 # will fail with package not found errors. Note also --assume-yes option which
 # shortcuts the interactive confirmation.
 #
+# Note that in this example custom commands will run after installing required
+# packages. If you have a PyPI package that depends on one of the custom
+# commands, move installation of the dependent package to the list of custom
+# commands, e.g.:
+#
+# ['pip', 'install', 'my_package'],
+#
+# TODO(BEAM-3237): Output from the custom commands are missing from the logs.
 # The output of custom commands (including failures) will be logged in the
 # worker-startup log.
 CUSTOM_COMMANDS = [

[GitHub] beam pull request #4167: Update julia set example comments

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4167


---

[jira] [Updated] (BEAM-3186) In-flight data loss when restoring from savepoint

[Pawel Bartoszek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Bartoszek updated BEAM-3186:
--
Attachment: restore_with_trigger_b.png

As [~aljoscha] suggested I performed another test depicted below:

!restore_with_trigger_b.png!

After restore I have got 2 B events but A events are lost

> In-flight data loss when restoring from savepoint
> -
>
> Key: BEAM-3186
> URL: https://issues.apache.org/jira/browse/BEAM-3186
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Aljoscha Krettek
> Attachments: restore_no_trigger.png, restore_with_trigger.png, 
> restore_with_trigger_b.png
>
>
> *The context:*
> I want to count how many events of given type(A,B, etc) I receive every 
> minute using 1 minute windows and AfterWatermark trigger with allowed 
> lateness 1 min.
> *Data loss case*
> In the case below if there is at least one A element with the event time 
> belonging to the window 14:00-14:01 read from Kinesis stream after job is 
> restored from savepoint the data loss will not be observed for this key and 
> this window.
> !restore_no_trigger.png!
> *Not data loss case*
> However, if no new A element element is read from Kinesis stream than data 
> loss is observable.
> !restore_with_trigger.png!
> *Workaround*
> As a workaround we could configure early firings every X seconds which gives 
> up to X seconds data loss per key on restore.
> *My guess where the issue might be*
> I believe this is Beam-Flink integration layer bug. From my investigation I 
> don't think it's KinesisReader and possibility that it couldn't advance 
> watermark. To prove that after I restore from savepoint I sent some records 
> for different key (B) for the same window as shown in the 
> pictures(14:00-14:01) without seeing trigger going off for restored window 
> and key A.
> My guess is that Beam after job is restored doesn't register flink event time 
> timer for restored window unless there is a new element (key) coming for the 
> restored window.
> Please refer to [this 
> gist|https://gist.github.com/pbartoszek/7ab88c8b6538039db1b383358d1d1b5a] for 
> test job that shows this behaviour.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263250#comment-16263250
 ] 

Valentyn Tymofieiev commented on BEAM-3120:
---

I looked at the staged version of the SDK - it includes changes from 
https://github.com/apache/beam/commit/3b24ce8d649c44fb4fa867b9706014852eeb41da. 
Which means it's no longer that 7 days old, but still more than 1 day old since 
 https://github.com/apache/beam/commit/fc57082ce76d0821b0b4979fe3d7935e2f8e64ae 
is not picked up.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-3120) Jenkins postcommit test suite triggered for a release branch uses code from master branch.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263250#comment-16263250
 ] 

Valentyn Tymofieiev edited comment on BEAM-3120 at 11/22/17 7:56 PM:
-

I looked at the staged version of the SDK - it includes changes from 
https://github.com/apache/beam/commit/3b24ce8d649c44fb4fa867b9706014852eeb41da. 
Which means it's no longer that 7 days old, but still old since  
https://github.com/apache/beam/commit/fc57082ce76d0821b0b4979fe3d7935e2f8e64ae 
is not picked up.


was (Author: tvalentyn):
I looked at the staged version of the SDK - it includes changes from 
https://github.com/apache/beam/commit/3b24ce8d649c44fb4fa867b9706014852eeb41da. 
Which means it's no longer that 7 days old, but still more than 1 day old since 
 https://github.com/apache/beam/commit/fc57082ce76d0821b0b4979fe3d7935e2f8e64ae 
is not picked up.

> Jenkins postcommit test suite triggered for a release branch uses code from 
> master branch.
> --
>
> Key: BEAM-3120
> URL: https://issues.apache.org/jira/browse/BEAM-3120
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>
> From 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/3452/
>  we can see the build was triggered on Revision: 
> 576d22a67ffeb29adf26f8adc09aaa6078099cf6
> origin/release-2.2.0. However, looking at test logs we can see that 2.3.0.dev 
> version of the Python SDK is being installed, and tested:
> # Tox runs unit tests in a virtual environment
> ${LOCAL_PATH}/tox -e ALL -c sdks/python/tox.ini
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/setup.py
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/docs
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> docs installed: 
> alabaster==0.7.10,apache-beam==2.3.0.dev0,avro==1.8.2,Babel==2.5.1,certifi==2017.7.27.1,chardet==3.0.4,crcmod==1.7,dill==0.2.6,docutils==0.14,enum34==1.1.6,funcsigs==1.0.2,futures==3.1.1,grpcio==1.7.0,grpcio-tools==1.3.5,httplib2==0.9.2,idna==2.6,imagesize==0.7.1,Jinja2==2.9.6,MarkupSafe==1.0,mock==2.0.0,nose==1.3.7,oauth2client==3.0.0,pbr==3.1.1,protobuf==3.3.0,pyasn1==0.3.7,pyasn1-modules==0.1.5,Pygments==2.2.0,pytz==2017.3,PyYAML==3.12,requests==2.18.4,rsa==3.4.2,six==1.10.0,snowballstemmer==1.2.1,Sphinx==1.5.5,sphinx-rtd-theme==0.2.4,typing==3.6.2,urllib3==1.22
> Looking further.
> cc [~altay]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4394

[Apache Jenkins Server]

See 

[GitHub] beam pull request #4167: Update julia set example comments

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/4167

Update julia set example comments

R: @charlesccychen 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam julia

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4167.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4167


commit 7fe7732bbf7721231ef7ec072307e533f006aef3
Author: Ahmet Altay 
Date:   2017-11-22T19:43:05Z

Update julia set example comments




---