[jira] [Work logged] (BEAM-4354) Enforce ErrorProne analysis in the tika IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4354?focusedWorklogId=104975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104975
 ]

ASF GitHub Bot logged work on BEAM-4354:


Author: ASF GitHub Bot
Created on: 23/May/18 04:31
Start Date: 23/May/18 04:31
Worklog Time Spent: 10m 
  Work Description: timrobertson100 opened a new pull request #5454: 
[BEAM-4354] Enforce ErrorProne analysis in tika IO
URL: https://github.com/apache/beam/pull/5454
 
 
   Enforces error prone. 
   Only warnings raised relate to missing annotations for the 
`package-info.class`. Note that `Tika` needs to bring in the `bndlib` in 
addition to to `findbugs_annotations` because it [declares it as provided in 
scope 
here](https://github.com/apache/tika/blob/33f3855bf36a0d119f77e0dbb3ccf1b8af5d082d/tika-java7/pom.xml#L90).
   
   CC @iemejia for a review please - hopefully a super simple one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104975)
Time Spent: 10m
Remaining Estimate: 0h

> Enforce ErrorProne analysis in the tika IO project
> --
>
> Key: BEAM-4354
> URL: https://issues.apache.org/jira/browse/BEAM-4354
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-tika
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-tika}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-tika:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/tika}} project.
> # In {{sdks/java/io/tika/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4387) [SQL] Implement date types comparisons

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-4387:
-

 Summary: [SQL] Implement date types comparisons
 Key: BEAM-4387
 URL: https://issues.apache.org/jira/browse/BEAM-4387
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin


implement and document datetime/timestamp etc comparisons



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3785) [SQL] Add support for arrays

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3785.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> [SQL] Add support for arrays
> 
>
> Key: BEAM-3785
> URL: https://issues.apache.org/jira/browse/BEAM-3785
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Support fields of Array type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3789) [SQL] Support Nested Rows

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3789.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> [SQL] Support Nested Rows
> -
>
> Key: BEAM-3789
> URL: https://issues.apache.org/jira/browse/BEAM-3789
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add support for SqlTypeName.ROW



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #324

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[ankurgoenka] Updating the dostring

--
[...truncated 18.03 MB...]
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 23, 2018 3:40:52 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 23, 2018 3:40:52 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523034046-cdd8ef59/output/results/staging/
May 23, 2018 3:40:52 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash _HN0Fa2MkFbUAiDaG-EYsA> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523034046-cdd8ef59/output/results/staging/pipeline-_HN0Fa2MkFbUAiDaG-EYsA.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 23, 2018 3:40:54 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 

[jira] [Closed] (BEAM-4162) Wire up PubsubIO+JSON to Beam SQL

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-4162.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Wire up PubsubIO+JSON to Beam SQL
> -
>
> Key: BEAM-4162
> URL: https://issues.apache.org/jira/browse/BEAM-4162
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Read JSON messages from Pubsub, convert them to Rows (BEAM-4160), wire up to 
> Beam SQL.
>  
> Use publication time as event timestamp



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4199) [SQL] Add a DLQ support for Pubsub tables

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-4199.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> [SQL] Add a DLQ support for Pubsub tables
> -
>
> Key: BEAM-4199
> URL: https://issues.apache.org/jira/browse/BEAM-4199
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Currently we crash the pipeline if there's any error processing the message 
> from the pubsub, including if it has incorrect JSON format, like missing 
> fields etc.
> Correct solution would be for the user to specify a way to handle the errors, 
> and ideally point to a dead-letter-queue where Beam should send the messages 
> it could not process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4386) [SQL] Implement string LIKE operator

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-4386.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> [SQL] Implement string LIKE operator
> 
>
> Key: BEAM-4386
> URL: https://issues.apache.org/jira/browse/BEAM-4386
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> Implent string LIKE operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4386) [SQL] Implement string LIKE operator

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-4386:
-

 Summary: [SQL] Implement string LIKE operator
 Key: BEAM-4386
 URL: https://issues.apache.org/jira/browse/BEAM-4386
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin


Implent string LIKE operator



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4385) Support LIKE operator

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-4385:
-

 Summary: Support LIKE operator
 Key: BEAM-4385
 URL: https://issues.apache.org/jira/browse/BEAM-4385
 Project: Beam
  Issue Type: New Feature
  Components: dsl-sql
Reporter: Kenneth Knowles


Currently the LIKE operator is not supported. It is pretty important.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104941
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 02:36
Start Date: 23/May/18 02:36
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391201647
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104941)
Time Spent: 3.5h  (was: 3h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104927
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 01:47
Start Date: 23/May/18 01:47
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190102635
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
+loggingServer.getApiServiceDescriptor(),
+controlServer.getApiServiceDescriptor(),
+InProcessManagedChannelFactory.create(),
+StreamObserverFactory.direct()));
+executor.submit(
+() -> {
+  try {
+fnHarness.get();
+  } catch (Throwable t) {
+executor.shutdownNow();
+  }
+});
+
+// TODO: find some way to populate the actual ID in FnHarness.main()
 
 Review comment:
   I'm not sure I understand the context for this question. This ID is passed 
as part of the container contract and used for all portable API calls from that 
container. I think it's desirable that it's a single ID for simplicity and 
debuggability. Python's use of an environment variable to pass it around 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #323

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Futurize internal subpackage

[matthias] Futurize options subpackage

--
[...truncated 18.15 MB...]
INFO: Adding OutputSideInputs as step s11
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 23, 2018 1:38:32 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 23, 2018 1:38:32 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523013827-b12703e9/output/results/staging/
May 23, 2018 1:38:32 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71120 bytes, hash gmsqi43JO3L_ZncYbhfxVA> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0523013827-b12703e9/output/results/staging/pipeline-gmsqi43JO3L_ZncYbhfxVA.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
May 23, 2018 1:38:34 AM 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #518

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104910
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095184
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlPrimitive.java
 ##
 @@ -111,54 +112,57 @@ public boolean accept() {
 }
 
 switch (outputType) {
-case BIGINT:
-  return value instanceof Long;
-case DECIMAL:
-  return value instanceof BigDecimal;
-case DOUBLE:
-  return value instanceof Double;
-case FLOAT:
-  return value instanceof Float;
-case INTEGER:
-  return value instanceof Integer;
-case SMALLINT:
-  return value instanceof Short;
-case TINYINT:
-  return value instanceof Byte;
-case BOOLEAN:
-  return value instanceof Boolean;
-case CHAR:
-case VARCHAR:
-  return value instanceof String || value instanceof NlsString;
-case TIME:
-  return value instanceof ReadableInstant;
-case TIMESTAMP:
-case DATE:
-  return value instanceof ReadableInstant;
-case INTERVAL_SECOND:
-case INTERVAL_MINUTE:
-case INTERVAL_HOUR:
-case INTERVAL_DAY:
-case INTERVAL_MONTH:
-case INTERVAL_YEAR:
-  return value instanceof BigDecimal;
-case SYMBOL:
-  // for SYMBOL, it supports anything...
-  return true;
-case ARRAY:
-  return value instanceof List;
-case MAP:
-  return value instanceof Map;
-case ROW:
-  return value instanceof Row;
-default:
-  throw new UnsupportedOperationException(
-  "Unsupported Beam SQL type in expression: " + outputType.name());
+  case BIGINT:
+return value instanceof Long;
+  case DECIMAL:
+return value instanceof BigDecimal;
+  case DOUBLE:
+return value instanceof Double;
+  case FLOAT:
+return value instanceof Float;
+  case INTEGER:
+return value instanceof Integer;
+  case SMALLINT:
+return value instanceof Short;
+  case TINYINT:
+return value instanceof Byte;
+  case BOOLEAN:
+return value instanceof Boolean;
+  case CHAR:
+  case VARCHAR:
+return value instanceof String || value instanceof NlsString;
+  case TIME:
+return value instanceof ReadableInstant;
+  case TIMESTAMP:
+  case DATE:
+return value instanceof ReadableInstant;
+  case INTERVAL_SECOND:
+  case INTERVAL_MINUTE:
+  case INTERVAL_HOUR:
+  case INTERVAL_DAY:
+  case INTERVAL_MONTH:
+  case INTERVAL_YEAR:
+return value instanceof BigDecimal;
+  case SYMBOL:
+// for SYMBOL, it supports anything...
+return true;
+  case ARRAY:
+return value instanceof List;
+  case MAP:
+return value instanceof Map;
+  case ROW:
+return value instanceof Row;
+  case MULTISET:
+return value instanceof Iterable;
 
 Review comment:
   Yes, I believe that is correct. It seems to make semantic sense - anything 
where you can get the values out of it, basically. And MULTISET is actually the 
least constrained of any collection (am I right?) so I think anything works.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104910)
Time Spent: 2h 20m  (was: 2h 10m)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104911
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095082
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
 ##
 @@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+  private final int correlationId;
+
+  public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int 
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+  }
+
+  @Override
+  public boolean accept() {
+return true;
+  }
+
+  @Override
+  public BeamSqlPrimitive evaluate(
+  Row inputRow, BoundedWindow window, ImmutableMap 
correlateEnv) {
 
 Review comment:
   Oh, I think that the correlation expression - this class - will work in 
general. It can be nested arbitrarily AFAIK. The place where that support is 
missing is at the `Rel` level. If we had a general `BeamCorrelateRel` then it 
would have to add its var to the environment just like the `BeamUnnestRel` 
does. And to be general, all of the `Rel` bits would also need to also know 
about the environment and do something clever. I'm not sure that would be the 
overall solution - likely we want to successfully use Calcite's decorrelate. It 
did not seem to actually decorrelate this particular query, though I might have 
just been using it wrong.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104911)
Time Spent: 2h 20m  (was: 2h 10m)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104912
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 23/May/18 00:46
Start Date: 23/May/18 00:46
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5428: [BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190095340
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
 ##
 @@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.List;
+import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionExecutor;
+import org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlFnExecutor;
+import org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Uncollect;
+import org.apache.calcite.sql.SemiJoinType;
+import org.apache.calcite.util.ImmutableBitSet;
+
+/**
+ * {@link BeamRelNode} to implement UNNEST, supporting specifically only 
{@link Correlate} with
+ * {@link Uncollect}.
+ */
+public class BeamUnnestRel extends Correlate implements BeamRelNode {
+
+  public BeamUnnestRel(
+  RelOptCluster cluster,
+  RelTraitSet traits,
+  RelNode left,
+  RelNode right,
+  CorrelationId correlationId,
+  ImmutableBitSet requiredColumns,
+  SemiJoinType joinType) {
+super(cluster, traits, left, right, correlationId, requiredColumns, 
joinType);
+  }
+
+  @Override
+  public PTransform toPTransform() {
+return new Transform();
+  }
+
+  @Override
+  public Correlate copy(
+  RelTraitSet relTraitSet,
+  RelNode left,
+  RelNode right,
+  CorrelationId correlationId,
+  ImmutableBitSet requireColumns,
+  SemiJoinType joinType) {
+return new BeamUnnestRel(
+getCluster(), relTraitSet, left, right, correlationId, 
requiredColumns, joinType);
+  }
+
+  private class Transform extends PTransform {
+@Override
+public PCollection expand(PCollectionTuple inputPCollections) {
+  String stageName = BeamSqlRelUtils.getStageName(BeamUnnestRel.this);
+
+  // The set of rows where we run the correlated unnest for each row
+  PCollection outer =
+  
inputPCollections.apply(BeamSqlRelUtils.getBeamRelInput(left).toPTransform());
+
+  // The correlated subquery
+  BeamUncollectRel uncollect = (BeamUncollectRel) 
BeamSqlRelUtils.getBeamRelInput(right);
+  Schema innerSchema = CalciteUtils.toBeamSchema(uncollect.getRowType());
+  checkArgument(
+  innerSchema.getFieldCount() == 1, "Can only UNNEST a single column", 
getClass());
+
+  BeamSqlExpressionExecutor expr =
+  new 
BeamSqlFnExecutor(BeamSqlRelUtils.getBeamRelInput(uncollect.getInput()));
+
+  Schema joinedSchema = CalciteUtils.toBeamSchema(rowType);
+
+  return outer
+  .apply(
+  stageName,
+  ParDo.of(
+  new UnnestFn(correlationId.getId(), 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104907
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:41
Start Date: 23/May/18 00:41
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190094767
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
 
 Review comment:
   Updated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104907)
Time Spent: 14h  (was: 13h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 14h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104906
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:41
Start Date: 23/May/18 00:41
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190094759
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104906)
Time Spent: 13h 50m  (was: 13h 40m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104908
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:41
Start Date: 23/May/18 00:41
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190094789
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
 
 Review comment:
   Nope, done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104908)
Time Spent: 14h 10m  (was: 14h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 14h 10m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104905
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:40
Start Date: 23/May/18 00:40
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190094746
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
 
 Review comment:
   I've added BEAM-4384;
   
   I don't think this is particularly high priority - I don't expect most users 
to use this directly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104905)
Time Spent: 13h 40m  (was: 13.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New 

[jira] [Created] (BEAM-4384) Load in-process FnHarness instances with a restricted classpath

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-4384:
-

 Summary: Load in-process FnHarness instances with a restricted 
classpath
 Key: BEAM-4384
 URL: https://issues.apache.org/jira/browse/BEAM-4384
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Reporter: Thomas Groh


The new classpath can contain the Java harness code and the user's classpath. 
This simulates the SDK container, and removes the need for users to provide 
beam-sdks-java-harness as a dependency.

The idea would be that beam-sdks-java-harness would be a jar containing all of 
its dependencies shaded away similar to what we do for Dataflow. We would embed 
the beam-sdks-java-harness.jar as a jar inside the direct runner jar so we 
wouldn't need to fetch it from Maven and also would be able to solve the 
problem where tests would run from the provided jar.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #517

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[ankurgoenka] Updating the dostring

--
[...truncated 18.89 MB...]
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 23, 2018 12:28:11 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-22_17_28_10-2758092648441612008?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-22_17_28_10-2758092648441612008

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 23, 2018 12:28:11 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-22_17_28_10-2758092648441612008
May 23, 2018 12:28:11 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-22_17_28_10-2758092648441612008 with 0 
expected assertions.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:10.188Z: Autoscaling is enabled for job 
2018-05-22_17_28_10-2758092648441612008. The number of workers will be between 
1 and 1000.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:10.227Z: Autoscaling was automatically enabled for 
job 2018-05-22_17_28_10-2758092648441612008.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:12.781Z: Checking required Cloud APIs are enabled.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:12.898Z: Checking permissions granted to controller 
Service Account.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:16.849Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.207Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.489Z: Expanding GroupByKey operations into 
optimizable parts.
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.536Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.836Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.904Z: Elided trivial flatten 
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.951Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:17.989Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 23, 2018 12:28:25 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-23T00:28:18.032Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104901
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:31
Start Date: 23/May/18 00:31
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190093623
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
 
 Review comment:
   It's fine, but also gets us very little. Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104901)
Time Spent: 13.5h  (was: 13h 20m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104900
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:31
Start Date: 23/May/18 00:31
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190093615
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
 ##
 @@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.Iterables;
+import java.io.Serializable;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.joda.time.Instant;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link RemoteStageEvaluatorFactory}. */
+@RunWith(JUnit4.class)
+public class RemoteStageEvaluatorFactoryTest implements Serializable {
+
+  private transient RemoteStageEvaluatorFactory factory;
+  private transient ExecutorService executor;
+  private transient GrpcFnServer dataServer;
 
 Review comment:
   We can't - we don't create the harness ourselves, we rely on the 
`EnvironmentFactory` to do so.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104900)
Time Spent: 13h 20m  (was: 13h 10m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104889=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104889
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:26
Start Date: 23/May/18 00:26
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190092945
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -58,12 +58,14 @@ dependencies {
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
   compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  testCompile project(path: ":beam-sdks-java-harness")
 
 Review comment:
   Now it's required because the `compileOnly` dependency is in a different 
project.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104889)
Time Spent: 13h 10m  (was: 13h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #202

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Futurize internal subpackage

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[matthias] Futurize options subpackage

[timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO

[timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO

[timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability

[ankurgoenka] Updating the dostring

--
[...truncated 408.62 KB...]
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1657)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.match(HadoopFileSystem.java:81)
at org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:123)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeleteFileFn.processElement(FileBasedIOITHelper.java:89)
at 
org.apache.beam.sdk.io.common.FileBasedIOITHelper$DeleteFileFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
avroioit0writethenreadall-05221704-b8x1-harness-j5jc.c.apache-beam-testing.internal/10.128.0.22
 to 71.211.226.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #202

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Futurize internal subpackage

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[matthias] Futurize options subpackage

[timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO

[timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO

[timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability

[ankurgoenka] Updating the dostring

--
[...truncated 545.00 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05221705-411s-harness-k0sv.c.apache-beam-testing.internal/10.128.0.2
 to 28.62.184.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #207

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Futurize internal subpackage

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[matthias] Futurize options subpackage

[timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO

[timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO

[timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability

[ankurgoenka] Updating the dostring

--
[...truncated 212.64 KB...]
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=104.197.134.90:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=104.197.134.90:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104886
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 23/May/18 00:11
Start Date: 23/May/18 00:11
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190090893
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -58,12 +58,14 @@ dependencies {
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
   compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
 
 Review comment:
   That's what this is, as far as I can tell - 
https://blog.gradle.org/introducing-compile-only-dependencies


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104886)
Time Spent: 13h  (was: 12h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #201

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #25

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Futurize internal subpackage

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[ankurgoenka] Adding ArtifactStagingFileHandler to push files to staging server

[ankurgoenka] Adding filesystem based artifact service to ULR job service

[ankurgoenka] Adding context manager to ArtifactStagingFileHandler

[ankurgoenka] Converting ArtifactServiceFileHandler to PortableStager based on

[ankurgoenka] Simplifying numeric constants

[matthias] Futurize options subpackage

[timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO

[timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO

[timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO

[ankurgoenka] Logging and naming changes

[ankurgoenka] Enhancing test cases

[timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability

[ankurgoenka] Updating the dostring

--
[...truncated 92.87 KB...]
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 13,5,main]) completed. Took 0.011 secs.

> Task :beam-sdks-java-core:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:compileTestJava' is 
df96519bfbc4048e225f237aaa685c4f
Skipping task ':beam-sdks-java-core:compileTestJava' as it is up-to-date.
:beam-sdks-java-core:compileTestJava (Thread[Task worker for ':' Thread 
5,5,main]) completed. Took 0.218 secs.
:beam-sdks-java-core:processTestResources (Thread[Task worker for ':' Thread 
5,5,main]) started.

> Task :beam-sdks-java-core:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-sdks-java-core:processTestResources' as it has no source 
files and no previous output files.
:beam-sdks-java-core:processTestResources (Thread[Task worker for ':' Thread 
5,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-core:testClasses (Thread[Task worker for ':' Thread 5,5,main]) 
started.

> Task :beam-sdks-java-core:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-core:testClasses' as it has no actions.
:beam-sdks-java-core:testClasses (Thread[Task worker for ':' Thread 5,5,main]) 
completed. Took 0.0 secs.
:beam-sdks-java-core:shadowTestJar (Thread[Task worker for ':' Thread 
5,5,main]) started.

> Task :beam-sdks-java-core:shadowTestJar UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:shadowTestJar' is 
88833967a2be6f0ee0c09646d6ea403f
Caching disabled for task ':beam-sdks-java-core:shadowTestJar': Caching has not 
been enabled for the task
Skipping task ':beam-sdks-java-core:shadowTestJar' as it is up-to-date.
:beam-sdks-java-core:shadowTestJar (Thread[Task worker for ':' Thread 
5,5,main]) completed. Took 0.023 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 5,5,main]) started.
:beam-sdks-java-core:jar (Thread[Task worker for ':' Thread 13,5,main]) started.

> Task :beam-sdks-java-core:jar UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:jar' is 
62194a91545dae98e776da16d0c3a4d7
Caching disabled for task ':beam-sdks-java-core:jar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-core:jar' as it is up-to-date.
:beam-sdks-java-core:jar (Thread[Task worker for ':' Thread 13,5,main]) 
completed. Took 0.01 secs.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' is 
d87d62731581e52c80e65d13b1fcd8d8
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 5,5,main]) completed. Took 0.03 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 5,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:testClasses 
> UP-TO-DATE
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:testClasses' as it has 
no actions.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 5,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
> UP-TO-DATE
Build cache key for task 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104884
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 23/May/18 00:02
Start Date: 23/May/18 00:02
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391178631
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104884)
Time Spent: 3h 20m  (was: 3h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104883
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:58
Start Date: 22/May/18 23:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391177810
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104883)
Time Spent: 3h 10m  (was: 3h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104876
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190084189
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
 
 Review comment:
   Weren't we going to merge these two, branching on type?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104876)
Time Spent: 40m  (was: 0.5h)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104877
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190085702
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
 
 Review comment:
   Order these messages the same as above. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104877)
Time Spent: 40m  (was: 0.5h)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104879
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190085529
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
 
 Review comment:
   For consistency, I would order them int, double, string. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104879)
Time Spent: 50m  (was: 40m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104880
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190085844
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
+  repeated double double_values = 2;
+}
+
+// Data associated with a distribution metric.
+// This is based off of the current DistributionData metric
+// This is not a stackdriver or dropwizard compatible
+// style of distribution metric.
+message DistributionData {
+  oneof distribution {
+IntDistributionData int_double_distribution = 1;
 
 Review comment:
   int_double_distribution vs. double_distribution_data?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104880)
Time Spent: 1h  (was: 50m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual 

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104882
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190088024
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
+}
+
+// Extrema messages are used for calculating
+// Top-N/Bottom-N metrics.
+message Extrema {
+  // Only one of the two should be specified.
+  // Note: oneof is not allowed on repeated fields.
+  repeated int64 int_values = 1;
+  repeated double double_values = 2;
 
 Review comment:
   Top and bottom strings makes sense as well. (Actually, one of the most 
useful extrema is MostFrequent). 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104882)
Time Spent: 1h 10m  (was: 1h)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104881
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190087423
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -257,6 +262,122 @@ message ProcessBundleProgressRequest {
   string instruction_reference = 1;
 }
 
+message MonitoringInfo {
+  // The name defining the metric or monitored state.
+  string urn = 1;
+
+  // This is specified as a URN that implies:
+  // A message class: (Distribution, Counter, Extrema, MonitoringDataTable).
+  // Sub types like field formats - int64, double, string.
+  // Aggregation methods - SUM, LATEST, TOP-N, BOTTOM-N, DISTRIBUTION
+  // valid values are:
+  // beam:metrics:[SumInt64|LatestInt64|Top-NInt64|Bottom-NInt64|
+  // SumDouble|LatestDouble|Top-NDouble|Bottom-NDouble|DistributionInt64|
+  // DistributionDouble|MonitoringDataTable]
+  string type = 2;
+
+  // The Metric or monitored state.
+  oneof monitoring_status {
+MonitoringTableData monitored_table_data = 3;
+Metric metric = 4;
+  }
+
+  // A set of key+value labels which define the scope of the metric.
+  // Either a well defined entity id for the keys:
+  // “transform”, “pcollection”, “windowing_strategy”,
+  // “coder”, “environment” or any arbitrary label
+  // set by a custom metric or user metric.
+  // A monitoring system is expected to be able to aggregate the metric 
together
+  // for all updates having the same URN and labels.
+  // Some systems such as Stackdriver will be able to aggregate the metric
+  // using a subset of the provided labels
+  map labels = 5;
+}
+
+message Metric {
+  // (Required) The data for this metric.
+  oneof data {
+CounterData counter_data = 1;
+DistributionData distribution_data = 2;
+Extrema extrema_data = 3;
+  }
+}
+
+// Data associated with a Counter or Gauge metric.
+// This is designed to be compatible with metric collection
+// systems such as DropWizard.
+message CounterData {
+   oneof value {
+ int64 int64_value = 1;
+ string string_value = 2;
+ double double_value = 3;
+   }
 
 Review comment:
   No LatestString?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104881)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3926) Support MetricsPusher in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3926?focusedWorklogId=104878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104878
 ]

ASF GitHub Bot logged work on BEAM-3926:


Author: ASF GitHub Bot
Created on: 22/May/18 23:54
Start Date: 22/May/18 23:54
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #5437: 
[BEAM-3926] Add new metrics protos based on "Defining and adding SDK Metrics" 
htt…
URL: https://github.com/apache/beam/pull/5437#discussion_r190083759
 
 

 ##
 File path: model/fn-execution/src/main/proto/beam_fn_api.proto
 ##
 @@ -239,13 +239,18 @@ message ProcessBundleRequest {
 
 // Stable
 message ProcessBundleResponse {
-  // (Optional) If metrics reporting is supported by the SDK, this represents
-  // the final metrics to record for this bundle.
-  Metrics metrics = 1;
+  // DEPRECATED. Please use monitoring_info.
+  Metrics metrics = 1 [deprecated=true];
 
   // (Optional) Specifies that the bundle has been split since the last
   // ProcessBundleProgressResponse was sent.
   BundleSplit split = 2;
+
+  // (Optional) If metrics or monitored sttate reporting is supported by
 
 Review comment:
   s/sttate/state


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104878)
Time Spent: 50m  (was: 40m)

> Support MetricsPusher in Dataflow Runner
> 
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> See [relevant email 
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
>  From [~echauchot]:
>   
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very 
> different from the others. It just submits a job to the cloud hosted engine. 
> So, no access to metrics container etc... from the runner. So I think that 
> the MetricsPusher (component responsible for merging metrics and pushing them 
> to a sink backend) must not be instanciated in DataflowRunner otherwise it 
> would be more a client (driver) piece of code and we will lose all the 
> interest of being close to the execution engine (among other things 
> instrumentation of the execution of the pipelines).  I think that the 
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104871
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190083099
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
 
 Review comment:
   The best scenario would be if we created a new classpath which contained the 
Java harness code + what we detect in the users classpath and ran that. This 
would have the benefit of simulating the SDK container and remove the need for 
users to remember to put provide beam-sdks-java-harness as a dependency. The 
idea would be that beam-sdks-java-harness would be a jar containing all of its 
dependencies shaded away similar to what we do for Dataflow. We would embed the 
beam-sdks-java-harness.jar as a jar inside the direct runner jar so we wouldn't 
need to fetch it from Maven and also would be able to solve the problem where 
tests would run from the provided jar.
   
   If this seems to much, a corresponding JIRA and updating this code so if the 
FnHarness.main would throw a `ClassNotFoundException`, we tell users that the 
shared process 

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104870
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190086882
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
 
 Review comment:
   Would it make sense to migrate the InProcessSdkHarness TestRule to use the 
InProcessEnvironmentFactory?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104870)
Time Spent: 12h  (was: 11h 50m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104875
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190082848
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
 
 Review comment:
   Shouldn't we propagate `PipelineOptions` through?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104875)
Time Spent: 12h 50m  (was: 12h 40m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104874
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190087507
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
 ##
 @@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.Iterables;
+import java.io.Serializable;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.joda.time.Instant;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link RemoteStageEvaluatorFactory}. */
+@RunWith(JUnit4.class)
+public class RemoteStageEvaluatorFactoryTest implements Serializable {
+
+  private transient RemoteStageEvaluatorFactory factory;
+  private transient ExecutorService executor;
+  private transient GrpcFnServer dataServer;
 
 Review comment:
   Use the InProcessSdkHarness TestRule 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104874)
Time Spent: 12h 40m  (was: 12.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104873
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190086276
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -58,12 +58,14 @@ dependencies {
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
   compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  testCompile project(path: ":beam-sdks-java-harness")
 
 Review comment:
   using `provided` scope above should allow you to remove this line here


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104873)
Time Spent: 12.5h  (was: 12h 20m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104868
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190082050
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
 
 Review comment:
   Is it excessively difficult to do this now?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104868)
Time Spent: 11h 40m  (was: 11.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104867
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190082730
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
 
 Review comment:
   The important point isn't that the in-process gRPC channel is used, it is 
that the runner, SDK, and users code and dependencies are all shared inside the 
same process.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104867)
Time Spent: 11.5h  (was: 11h 20m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104869
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190085434
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/InProcessEnvironmentFactory.java
 ##
 @@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.time.Duration;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
+import 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
+import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.sdk.fn.stream.StreamObserverFactory;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+/**
+ * An {@link EnvironmentFactory} that communicates to a {@link FnHarness} via 
the in-process gRPC
+ * channel.
+ *
+ * TODO: Move this class to the runners/java-fn-execution module, with the 
Java SDK Harness as a
+ * provided dependency.
+ */
+class InProcessEnvironmentFactory implements EnvironmentFactory {
+
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer controlServer;
+
+  private final ControlClientPool.Source clientSource;
+
+  InProcessEnvironmentFactory(
+  GrpcFnServer loggingServer,
+  GrpcFnServer controlServer,
+  ControlClientPool.Source clientSource) {
+this.loggingServer = loggingServer;
+this.controlServer = controlServer;
+checkArgument(
+loggingServer.getApiServiceDescriptor() != null,
+"Logging Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+checkArgument(
+controlServer.getApiServiceDescriptor() != null,
+"Control Server cannot have a null %s",
+ApiServiceDescriptor.class.getSimpleName());
+this.clientSource = clientSource;
+  }
+
+  @Override
+  public RemoteEnvironment createEnvironment(Environment container) throws 
Exception {
+ExecutorService executor = Executors.newSingleThreadExecutor();
+Future fnHarness =
+executor.submit(
+() ->
+FnHarness.main(
+PipelineOptionsFactory.create(),
+loggingServer.getApiServiceDescriptor(),
+controlServer.getApiServiceDescriptor(),
+InProcessManagedChannelFactory.create(),
+StreamObserverFactory.direct()));
+executor.submit(
+() -> {
+  try {
+fnHarness.get();
+  } catch (Throwable t) {
+executor.shutdownNow();
+  }
+});
+
+// TODO: find some way to populate the actual ID in FnHarness.main()
 
 Review comment:
   Instead of relying on the WORKER_ID environment variable in Python, we could 
make the client id as part of the ApiServiceDescriptor. This would make the 
client id chosen by whomever vends the client and removes looking up a property.
   
   @herohde @angoenka What do you think?

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=104872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104872
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 22/May/18 23:52
Start Date: 22/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5349: 
[BEAM-3326] Remote stage evaluator
URL: https://github.com/apache/beam/pull/5349#discussion_r190086221
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -58,12 +58,14 @@ dependencies {
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
   compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
+  compileOnly project(path: ":beam-sdks-java-harness")
 
 Review comment:
   use the `provided` scope.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104872)
Time Spent: 12h 20m  (was: 12h 10m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104866
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:45
Start Date: 22/May/18 23:45
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391175624
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104866)
Time Spent: 3h  (was: 2h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=104863=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104863
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 22/May/18 23:41
Start Date: 22/May/18 23:41
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #5273: [BEAM-3883] Adding 
Client to push artifacts to artifact staging service
URL: https://github.com/apache/beam/pull/5273
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/portability/portable_stager.py 
b/sdks/python/apache_beam/runners/portability/portable_stager.py
new file mode 100644
index 000..7113a251f24
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/portable_stager.py
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""A :class:`FileHandler` to work with :class:`ArtifactStagingServiceStub`.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+
+from apache_beam.portability.api import beam_artifact_api_pb2
+from apache_beam.portability.api import beam_artifact_api_pb2_grpc
+from apache_beam.runners.portability.stager import Stager
+
+
+class PortableStager(Stager):
+  """An implementation of :class:`Stager` to stage files on
+  ArtifactStagingService.
+
+  The class keeps track of pushed files and commit manifest once all files are
+  uploaded.
+
+  Note: This class is not thread safe and user of this class should ensure
+  thread safety.
+  """
+
+  def __init__(self, artifact_service_channel):
+"""Creates a new Stager to stage file to ArtifactStagingService.
+
+Args:
+  artifact_service_channel: Channel used to interact with
+ArtifactStagingService.User owns the channel and should close it when
+finished.
+"""
+super(PortableStager, self).__init__()
+self._artifact_staging_stub = beam_artifact_api_pb2_grpc.\
+ArtifactStagingServiceStub(channel=artifact_service_channel)
+self._artifacts = []
+
+  def stage_artifact(self, local_path_to_artifact, artifact_name):
+"""Stage a file to ArtifactStagingService.
+
+Args:
+  local_path_to_artifact: Path of file to be uploaded.
+  artifact_name: File name on the artifact server.
+"""
+if not os.path.isfile(local_path_to_artifact):
+  raise ValueError(
+  'Cannot stage {0} to artifact server. Only local files can be 
staged.'
+  .format(local_path_to_artifact))
+
+def artifact_request_generator():
+  metadata = beam_artifact_api_pb2.ArtifactMetadata(name=artifact_name)
+  request = beam_artifact_api_pb2.PutArtifactRequest(metadata=metadata)
+  yield request
+  with open(local_path_to_artifact, 'rb') as f:
+while True:
+  chunk = f.read(1 << 21)  # 2MB
+  if not chunk:
+break
+  request = beam_artifact_api_pb2.PutArtifactRequest(
+  data=beam_artifact_api_pb2.ArtifactChunk(data=chunk))
+  yield request
+  self._artifacts.append(metadata)
+
+self._artifact_staging_stub.PutArtifact(artifact_request_generator())
+
+  def commit_manifest(self):
+manifest = beam_artifact_api_pb2.Manifest(artifact=self._artifacts)
+self._artifacts = []
+self._artifact_staging_stub.CommitManifest(
+beam_artifact_api_pb2.CommitManifestRequest(manifest=manifest))
diff --git 
a/sdks/python/apache_beam/runners/portability/portable_stager_test.py 
b/sdks/python/apache_beam/runners/portability/portable_stager_test.py
new file mode 100644
index 000..181007de5f0
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/portable_stager_test.py
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work 

[beam] 01/01: Merge pull request #5273: [BEAM-3883] Adding Client to push artifacts to artifact staging service

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dd8dcc65f39f60004eb5df0c43f083a7d78e19c6
Merge: 15600d6 a0ec87c
Author: Eugene Kirpichov 
AuthorDate: Tue May 22 16:41:33 2018 -0700

Merge pull request #5273: [BEAM-3883] Adding Client to push artifacts to 
artifact staging service

[BEAM-3883] Adding Client to push artifacts to artifact staging service

 .../runners/portability/portable_stager.py |  86 +++
 .../runners/portability/portable_stager_test.py| 162 +
 2 files changed, 248 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104862
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:40
Start Date: 22/May/18 23:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391174800
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104862)
Time Spent: 2h 50m  (was: 2h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (15600d6 -> dd8dcc6)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 15600d6  Merge pull request #5334 from RobbeSneyders/internal
 add 67c97c9  Adding ArtifactStagingFileHandler to push files to staging 
server
 add 6a8650a  Adding filesystem based artifact service to ULR job service
 add acfa20d  Adding context manager to ArtifactStagingFileHandler
 add ea525b0  Converting ArtifactServiceFileHandler to PortableStager based 
on discussion in parent PR
 add 0d60ae4  Simplifying numeric constants
 add 7c33a70  Logging and naming changes
 add 4bc2006  Enhancing test cases
 add a0ec87c  Updating the dostring
 new dd8dcc6  Merge pull request #5273: [BEAM-3883] Adding Client to push 
artifacts to artifact staging service

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/portability/portable_stager.py |  86 +++
 .../runners/portability/portable_stager_test.py| 162 +
 2 files changed, 248 insertions(+)
 create mode 100644 
sdks/python/apache_beam/runners/portability/portable_stager.py
 create mode 100644 
sdks/python/apache_beam/runners/portability/portable_stager_test.py

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104859
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 23:40
Start Date: 22/May/18 23:40
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190085785
 
 

 ##
 File path: runners/local-artifact-service-java/pom.xml
 ##
 @@ -1,137 +0,0 @@
-
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104859)
Time Spent: 2h 50m  (was: 2h 40m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104860
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 23:40
Start Date: 22/May/18 23:40
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190085558
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -57,12 +57,14 @@ dependencies {
   compile project(path: ":beam-runners-core-java", configuration: "shadow")
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
+  compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  shadow library.java.args4j
 
 Review comment:
   It's not a dependency of that module.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104860)
Time Spent: 3h  (was: 2h 50m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104861
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 23:40
Start Date: 22/May/18 23:40
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190085807
 
 

 ##
 File path: settings.gradle
 ##
 @@ -50,8 +50,6 @@ include "beam-runners-google-cloud-dataflow-java"
 project(":beam-runners-google-cloud-dataflow-java").dir = 
file("runners/google-cloud-dataflow-java")
 include "beam-runners-java-fn-execution"
 project(":beam-runners-java-fn-execution").dir = 
file("runners/java-fn-execution")
-include "beam-local-artifact-service-java"
-project(":beam-local-artifact-service-java").dir = 
file("runners/local-artifact-service-java")
 include "beam-runners-local-java-core"
 project(":beam-runners-local-java-core").dir = file("runners/local-java")
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104861)
Time Spent: 3h 10m  (was: 3h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #322

[Apache Jenkins Server]

See 


Changes:

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

[timrobertson100] [BEAM-4348] Enforce ErrorProne analysis in kinesis IO

[timrobertson100] [BEAM-4349] Enforce ErrorProne analysis in mongodb IO

[timrobertson100] [BEAM-4351] Enforce ErrorProne analysis in mqtt IO

[timrobertson100] [BEAM-4348] Removes double-checked locking to aid readability

--
[...truncated 19.45 MB...]
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 22, 2018 11:32:39 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 22, 2018 11:32:39 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0522233233-7aa9d27f/output/results/staging/
May 22, 2018 11:32:39 PM 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104848
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:28
Start Date: 22/May/18 23:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391172892
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104848)
Time Spent: 2h 40m  (was: 2.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104846
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:20
Start Date: 22/May/18 23:20
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391171509
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104846)
Time Spent: 2.5h  (was: 2h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104845
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 23:09
Start Date: 22/May/18 23:09
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391169583
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104845)
Time Spent: 2h 20m  (was: 2h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104841
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 22:36
Start Date: 22/May/18 22:36
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391162438
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104841)
Time Spent: 2h 10m  (was: 2h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104838
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 22:23
Start Date: 22/May/18 22:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391159903
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104838)
Time Spent: 2h  (was: 1h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3999) Futurize and fix python 2 compatibility for internal subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3999?focusedWorklogId=104836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104836
 ]

ASF GitHub Bot logged work on BEAM-3999:


Author: ASF GitHub Bot
Created on: 22/May/18 22:12
Start Date: 22/May/18 22:12
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5334: [BEAM-3999] Futurize 
internal subpackage
URL: https://github.com/apache/beam/pull/5334
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/internal/__init__.py 
b/sdks/python/apache_beam/internal/__init__.py
index 0bce5d68f72..84381ed71d4 100644
--- a/sdks/python/apache_beam/internal/__init__.py
+++ b/sdks/python/apache_beam/internal/__init__.py
@@ -16,3 +16,5 @@
 #
 
 """For internal use only; no backwards-compatibility guarantees."""
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/internal/gcp/__init__.py 
b/sdks/python/apache_beam/internal/gcp/__init__.py
index 0bce5d68f72..84381ed71d4 100644
--- a/sdks/python/apache_beam/internal/gcp/__init__.py
+++ b/sdks/python/apache_beam/internal/gcp/__init__.py
@@ -16,3 +16,5 @@
 #
 
 """For internal use only; no backwards-compatibility guarantees."""
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/internal/gcp/auth.py 
b/sdks/python/apache_beam/internal/gcp/auth.py
index 97a0ef83e85..cff88879d1a 100644
--- a/sdks/python/apache_beam/internal/gcp/auth.py
+++ b/sdks/python/apache_beam/internal/gcp/auth.py
@@ -17,17 +17,19 @@
 
 """Dataflow credentials and authentication."""
 
+from __future__ import absolute_import
+
 import datetime
 import json
 import logging
 import os
 
+from future.moves.urllib.request import Request
+from future.moves.urllib.request import urlopen
 from oauth2client.client import GoogleCredentials
 from oauth2client.client import OAuth2Credentials
 
 from apache_beam.utils import retry
-from six.moves.urllib.request import Request
-from six.moves.urllib.request import urlopen
 
 # When we are running in GCE, we can authenticate with VM credentials.
 is_running_in_gce = False
diff --git a/sdks/python/apache_beam/internal/gcp/json_value.py 
b/sdks/python/apache_beam/internal/gcp/json_value.py
index c4f3d7ba4da..a4c6fb9d135 100644
--- a/sdks/python/apache_beam/internal/gcp/json_value.py
+++ b/sdks/python/apache_beam/internal/gcp/json_value.py
@@ -17,6 +17,10 @@
 
 """JSON conversion utility functions."""
 
+from __future__ import absolute_import
+
+from apache_beam.options.value_provider import ValueProvider
+
 # Protect against environments where apitools library is not available.
 # pylint: disable=wrong-import-order, wrong-import-position
 try:
@@ -25,9 +29,12 @@
   extra_types = None
 # pylint: enable=wrong-import-order, wrong-import-position
 
-import six
-
-from apache_beam.options.value_provider import ValueProvider
+try:  # Python 2
+  unicode # pylint: disable=unicode-builtin
+  long# pylint: disable=long-builtin
+except NameError: # Python 3
+  unicode = str
+  long = int
 
 _MAXINT64 = (1 << 63) - 1
 _MININT64 = - (1 << 63)
@@ -49,7 +56,7 @@ def get_typed_value_descriptor(obj):
 ~exceptions.TypeError: if the Python object has a type that is not
   supported.
   """
-  if isinstance(obj, six.string_types):
+  if isinstance(obj, (str, unicode)):
 type_name = 'Text'
   elif isinstance(obj, bool):
 type_name = 'Boolean'
@@ -101,11 +108,11 @@ def to_json_value(obj, with_type=False):
 return extra_types.JsonValue(object_value=json_object)
   elif with_type:
 return to_json_value(get_typed_value_descriptor(obj), with_type=False)
-  elif isinstance(obj, six.string_types):
+  elif isinstance(obj, (str, unicode)):
 return extra_types.JsonValue(string_value=obj)
   elif isinstance(obj, bool):
 return extra_types.JsonValue(boolean_value=obj)
-  elif isinstance(obj, six.integer_types):
+  elif isinstance(obj, (int, long)):
 if _MININT64 <= obj <= _MAXINT64:
   return extra_types.JsonValue(integer_value=obj)
 else:
diff --git a/sdks/python/apache_beam/internal/gcp/json_value_test.py 
b/sdks/python/apache_beam/internal/gcp/json_value_test.py
index c22d067beea..e7cf7f15e36 100644
--- a/sdks/python/apache_beam/internal/gcp/json_value_test.py
+++ b/sdks/python/apache_beam/internal/gcp/json_value_test.py
@@ -17,6 +17,8 @@
 
 """Unit tests for the json_value module."""
 
+from __future__ import absolute_import
+
 import unittest
 
 from apache_beam.internal.gcp.json_value import from_json_value
diff --git a/sdks/python/apache_beam/internal/module_test.py 
b/sdks/python/apache_beam/internal/module_test.py
index 

[beam] 01/01: Merge pull request #5334 from RobbeSneyders/internal

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 15600d612e9f7dcc060fd2ad88e7f0ee2167eeb1
Merge: f53d272 f46f440
Author: Ahmet Altay 
AuthorDate: Tue May 22 15:12:18 2018 -0700

Merge pull request #5334 from RobbeSneyders/internal

[BEAM-3999] Futurize internal subpackage

 sdks/python/apache_beam/internal/__init__.py  |  2 ++
 sdks/python/apache_beam/internal/gcp/__init__.py  |  2 ++
 sdks/python/apache_beam/internal/gcp/auth.py  |  6 --
 sdks/python/apache_beam/internal/gcp/json_value.py| 19 +--
 .../apache_beam/internal/gcp/json_value_test.py   |  2 ++
 sdks/python/apache_beam/internal/module_test.py   |  3 +++
 sdks/python/apache_beam/internal/pickler.py   |  2 ++
 sdks/python/apache_beam/internal/pickler_test.py  |  2 ++
 sdks/python/apache_beam/internal/util.py  |  6 ++
 sdks/python/apache_beam/internal/util_test.py |  1 +
 sdks/python/tox.ini   |  1 +
 11 files changed, 38 insertions(+), 8 deletions(-)

diff --cc sdks/python/tox.ini
index 82e8d07,481c5f9..3b33437
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@@ -99,8 -99,8 +99,9 @@@ deps 
flake8==3.5.0
  modules =
apache_beam/coders
+   apache_beam/internal
apache_beam/metrics
 +  apache_beam/options
  commands =
python --version
pip --version

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (f53d272 -> 15600d6)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f53d272  Merge pull request #5339 from Fematich/options
 add f46f440  Futurize internal subpackage
 new 15600d6  Merge pull request #5334 from RobbeSneyders/internal

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/internal/__init__.py  |  2 ++
 sdks/python/apache_beam/internal/gcp/__init__.py  |  2 ++
 sdks/python/apache_beam/internal/gcp/auth.py  |  6 --
 sdks/python/apache_beam/internal/gcp/json_value.py| 19 +--
 .../apache_beam/internal/gcp/json_value_test.py   |  2 ++
 sdks/python/apache_beam/internal/module_test.py   |  3 +++
 sdks/python/apache_beam/internal/pickler.py   |  2 ++
 sdks/python/apache_beam/internal/pickler_test.py  |  2 ++
 sdks/python/apache_beam/internal/util.py  |  6 ++
 sdks/python/apache_beam/internal/util_test.py |  1 +
 sdks/python/tox.ini   |  1 +
 11 files changed, 38 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #514

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104834
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 22:07
Start Date: 22/May/18 22:07
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5370: [BEAM-2885] Move the 
ReferenceRunner Job Service, Artifact Services to the DirectRunner module
URL: https://github.com/apache/beam/pull/5370#issuecomment-391156501
 
 
   The Job Server proper, as implemented here, is probably going to be ~100 
lines of code that's pretty tightly integrated with the `PortableDirectRunner`. 
If/when we have a more abstract implementation, the DirectRunner can migrate to 
use that interface, but the current implementation is basically only going to 
exercise ~5 methods on the `PortableDirectRunner` which don't have a coherent 
interface.
   
   That might be a good followup, though.
   
   I would generally not expect to see the `LocalArtifactServer` be reused.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104834)
Time Spent: 2h 40m  (was: 2.5h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5339 from Fematich/options

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f53d27275302c7891988c103d76cb1a56d43aafb
Merge: 30fa605 bae2edd
Author: Ahmet Altay 
AuthorDate: Tue May 22 15:06:38 2018 -0700

Merge pull request #5339 from Fematich/options

[BEAM-4002] Futurize options subpackage

 sdks/python/apache_beam/options/__init__.py| 1 +
 sdks/python/apache_beam/options/pipeline_options.py| 7 +--
 sdks/python/apache_beam/options/pipeline_options_test.py   | 2 ++
 sdks/python/apache_beam/options/pipeline_options_validator.py  | 3 +++
 sdks/python/apache_beam/options/pipeline_options_validator_test.py | 3 +++
 sdks/python/apache_beam/options/value_provider.py  | 6 ++
 sdks/python/apache_beam/options/value_provider_test.py | 3 +++
 sdks/python/tox.ini| 1 +
 8 files changed, 24 insertions(+), 2 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-4002) Futurize and fix python 2 compatibility for options subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4002?focusedWorklogId=104833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104833
 ]

ASF GitHub Bot logged work on BEAM-4002:


Author: ASF GitHub Bot
Created on: 22/May/18 22:06
Start Date: 22/May/18 22:06
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5339: [BEAM-4002] Futurize 
options subpackage
URL: https://github.com/apache/beam/pull/5339
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/options/__init__.py 
b/sdks/python/apache_beam/options/__init__.py
index cce3acad34a..f4f43cbb123 100644
--- a/sdks/python/apache_beam/options/__init__.py
+++ b/sdks/python/apache_beam/options/__init__.py
@@ -14,3 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index b5f9d77617d..0c96325d70c 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -17,7 +17,10 @@
 
 """Pipeline options obtained from command line parsing."""
 
+from __future__ import absolute_import
+
 import argparse
+from builtins import object
 
 from apache_beam.options.value_provider import RuntimeValueProvider
 from apache_beam.options.value_provider import StaticValueProvider
@@ -175,7 +178,7 @@ def from_dictionary(cls, options):
   A PipelineOptions object representing the given arguments.
 """
 flags = []
-for k, v in options.iteritems():
+for k, v in options.items():
   if isinstance(v, bool):
 if v:
   flags.append('--%s' % k)
@@ -233,7 +236,7 @@ def _visible_option_list(self):
   for option in dir(self._visible_options) if option[0] != '_')
 
   def __dir__(self):
-return sorted(dir(type(self)) + self.__dict__.keys() +
+return sorted(dir(type(self)) + list(self.__dict__.keys()) +
   self._visible_option_list())
 
   def __getattr__(self, name):
diff --git a/sdks/python/apache_beam/options/pipeline_options_test.py 
b/sdks/python/apache_beam/options/pipeline_options_test.py
index 66c69bdfc70..289e67f15fe 100644
--- a/sdks/python/apache_beam/options/pipeline_options_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_test.py
@@ -17,6 +17,8 @@
 
 """Unit tests for the pipeline options module."""
 
+from __future__ import absolute_import
+
 import logging
 import unittest
 
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator.py 
b/sdks/python/apache_beam/options/pipeline_options_validator.py
index 24d2e55eadb..874e739fb61 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator.py
@@ -19,7 +19,10 @@
 
 For internal use only; no backwards-compatibility guarantees.
 """
+from __future__ import absolute_import
+
 import re
+from builtins import object
 
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import DebugOptions
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator_test.py 
b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
index 8ff66c78f17..c0582e2f7d1 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
@@ -17,8 +17,11 @@
 
 """Unit tests for the pipeline options validator module."""
 
+from __future__ import absolute_import
+
 import logging
 import unittest
+from builtins import object
 
 from hamcrest.core.base_matcher import BaseMatcher
 
diff --git a/sdks/python/apache_beam/options/value_provider.py 
b/sdks/python/apache_beam/options/value_provider.py
index 4aaa0f74474..0ef83e344c9 100644
--- a/sdks/python/apache_beam/options/value_provider.py
+++ b/sdks/python/apache_beam/options/value_provider.py
@@ -19,6 +19,9 @@
 and dynamically provided values.
 """
 
+from __future__ import absolute_import
+
+from builtins import object
 from functools import wraps
 
 from apache_beam import error
@@ -66,6 +69,9 @@ def __eq__(self, other):
 return True
 return False
 
+  def __hash__(self):
+return hash((type(self), self.value_type, self.value))
+
 
 class RuntimeValueProvider(ValueProvider):
   runtime_options = None
diff --git a/sdks/python/apache_beam/options/value_provider_test.py 
b/sdks/python/apache_beam/options/value_provider_test.py
index 482ea47504a..bc027148234 100644
--- 

[beam] branch master updated (30fa605 -> f53d272)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 30fa605  Merge pull request #5448: [BEAM-4348] Enforce ErrorProne 
analysis in kinesis IO
 add bae2edd  Futurize options subpackage
 new f53d272  Merge pull request #5339 from Fematich/options

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/options/__init__.py| 1 +
 sdks/python/apache_beam/options/pipeline_options.py| 7 +--
 sdks/python/apache_beam/options/pipeline_options_test.py   | 2 ++
 sdks/python/apache_beam/options/pipeline_options_validator.py  | 3 +++
 sdks/python/apache_beam/options/pipeline_options_validator_test.py | 3 +++
 sdks/python/apache_beam/options/value_provider.py  | 6 ++
 sdks/python/apache_beam/options/value_provider_test.py | 3 +++
 sdks/python/tox.ini| 1 +
 8 files changed, 24 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104831
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 22:01
Start Date: 22/May/18 22:01
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391155079
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104831)
Time Spent: 1h 50m  (was: 1h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104822
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190064795
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java
 ##
 @@ -1,22 +1,25 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104822)
Time Spent: 1h 50m  (was: 1h 40m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104821
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190064994
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/package-info.java
 ##
 @@ -6,18 +6,18 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104821)
Time Spent: 1h 40m  (was: 1.5h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104830
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190066187
 
 

 ##
 File path: runners/local-artifact-service-java/pom.xml
 ##
 @@ -1,137 +0,0 @@
-
 
 Review comment:
   remove `local-artifact-service-java` from `runners/pom.xml`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104830)
Time Spent: 2.5h  (was: 2h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104825
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065417
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java
 ##
 @@ -6,17 +6,17 @@
  * to you under the Apache License, Version 2.0 (the
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
- *
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104825)
Time Spent: 2h 10m  (was: 2h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104823
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065380
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java
 ##
 @@ -1,22 +1,25 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104823)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104820
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190064483
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java
 ##
 @@ -0,0 +1,23 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
+ *  * with the License.  You may obtain a copy of the License at
 
 Review comment:
   ditto


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104820)
Time Spent: 1.5h  (was: 1h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104819
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190064453
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
 ##
 @@ -1,22 +1,25 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104819)
Time Spent: 1.5h  (was: 1h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104829
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190064911
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
 ##
 @@ -14,10 +14,11 @@
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
+ *
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104829)
Time Spent: 2.5h  (was: 2h 20m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104824
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065075
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
 ##
 @@ -1,22 +1,25 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
+ *  * regarding copyright ownership.  The ASF licenses this file
+ *  * to you under the Apache License, Version 2.0 (the
+ *  * "License"); you may not use this file except in compliance
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104824)
Time Spent: 2h  (was: 1h 50m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104828=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104828
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065176
 
 

 ##
 File path: runners/direct-java/build.gradle
 ##
 @@ -57,12 +57,14 @@ dependencies {
   compile project(path: ":beam-runners-core-java", configuration: "shadow")
   compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
+  compile project(path: ":beam-sdks-java-fn-execution", configuration: 
"shadow")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
   shadow library.java.hamcrest_core
   shadow library.java.junit
+  shadow library.java.args4j
 
 Review comment:
   should we remove args4j from reference runner as a dependency?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104828)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104826
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065786
 
 

 ##
 File path: settings.gradle
 ##
 @@ -50,8 +50,6 @@ include "beam-runners-google-cloud-dataflow-java"
 project(":beam-runners-google-cloud-dataflow-java").dir = 
file("runners/google-cloud-dataflow-java")
 include "beam-runners-java-fn-execution"
 project(":beam-runners-java-fn-execution").dir = 
file("runners/java-fn-execution")
-include "beam-local-artifact-service-java"
-project(":beam-local-artifact-service-java").dir = 
file("runners/local-artifact-service-java")
 include "beam-runners-local-java-core"
 project(":beam-runners-local-java-core").dir = file("runners/local-java")
 
 Review comment:
   remove `beam-runners-reference-job-server`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104826)
Time Spent: 2h 10m  (was: 2h)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2885) Support job+artifact APIs locally

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?focusedWorklogId=104827=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104827
 ]

ASF GitHub Bot logged work on BEAM-2885:


Author: ASF GitHub Bot
Created on: 22/May/18 21:59
Start Date: 22/May/18 21:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5370: 
[BEAM-2885] Move the ReferenceRunner Job Service, Artifact Services to the 
DirectRunner module
URL: https://github.com/apache/beam/pull/5370#discussion_r190065398
 
 

 ##
 File path: 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java
 ##
 @@ -1,22 +1,25 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  * Licensed to the Apache Software Foundation (ASF) under one
+ *  * or more contributor license agreements.  See the NOTICE file
+ *  * distributed with this work for additional information
 
 Review comment:
   revert license header change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104827)
Time Spent: 2h 20m  (was: 2h 10m)

> Support job+artifact APIs locally
> -
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=104818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104818
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 22/May/18 21:57
Start Date: 22/May/18 21:57
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-391153970
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104818)
Time Spent: 1h 40m  (was: 1.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4167) Implement UNNEST

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4167?focusedWorklogId=104815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104815
 ]

ASF GitHub Bot logged work on BEAM-4167:


Author: ASF GitHub Bot
Created on: 22/May/18 21:44
Start Date: 22/May/18 21:44
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #5428: 
[BEAM-4167] Implement UNNEST
URL: https://github.com/apache/beam/pull/5428#discussion_r190063251
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlCorrelVariableExpression.java
 ##
 @@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
+
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.Row;
+import org.apache.calcite.sql.type.SqlTypeName;
+
+/** A primitive operation for deferencing a correlation variable. */
+public class BeamSqlCorrelVariableExpression extends BeamSqlExpression {
+
+  private final int correlationId;
+
+  public BeamSqlCorrelVariableExpression(SqlTypeName sqlTypeName, int 
correlationId) {
+super(null, sqlTypeName);
+this.correlationId = correlationId;
+  }
+
+  @Override
+  public boolean accept() {
+return true;
+  }
+
+  @Override
+  public BeamSqlPrimitive evaluate(
+  Row inputRow, BoundedWindow window, ImmutableMap 
correlateEnv) {
 
 Review comment:
   I see. Would it work for complex cases, like multiple nested correlated 
queries? What the plan would look like? I don't see where this is handled, 
maybe I missed it.
   
   If not, then it probably makes sense to comment on correlation logic that 
it's only there to support unnest and is limited. And is there a need for a map 
in this case? Wouldn't a single id work?
   
   And should we wrap the map into some kind of execution context class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104815)
Time Spent: 2h 10m  (was: 2h)

> Implement UNNEST
> 
>
> Key: BEAM-4167
> URL: https://issues.apache.org/jira/browse/BEAM-4167
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We need to be able to convert collections to relations in the query to 
> perform any meaningful operations on them. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=104794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104794
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 22/May/18 21:25
Start Date: 22/May/18 21:25
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request 
#5443: [BEAM-4357] [SQL] Fix shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#discussion_r190058260
 
 

 ##
 File path: sdks/java/extensions/sql/build.gradle
 ##
 @@ -45,23 +48,27 @@ dependencies {
   compile "org.apache.calcite:calcite-core:$calcite_version"
   compile "org.apache.calcite:calcite-linq4j:$calcite_version"
   compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
-  compile "com.alibaba:fastjson:1.2.12"
   compileOnly library.java.findbugs_jsr305
   compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow project(path: ":beam-sdks-java-extensions-join-library", 
configuration: "shadow")
   shadow library.java.slf4j_api
+  shadow library.java.slf4j_jdk14
   shadow library.java.commons_csv
+  shadow library.java.commons_lang3
+  shadow library.java.jackson_databind
   shadow library.java.joda_time
+  shadow "com.alibaba:fastjson:1.2.12"
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-kafka", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")
   provided library.java.kafka_clients
-  testCompile library.java.slf4j_jdk14
   testCompile library.java.junit
   testCompile library.java.hamcrest_core
   testCompile library.java.mockito_core
   testCompile library.java.quickcheck_core
+  shadowJarTest project(path: project.path, configuration: "shadow")
 
 Review comment:
   Yea, I think we should do that. I saw that as one of the big correctness 
benefits of Gradle.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104794)
Time Spent: 1h 40m  (was: 1.5h)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=104793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104793
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 22/May/18 21:24
Start Date: 22/May/18 21:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5443: 
[BEAM-4357] [SQL] Fix shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#discussion_r190057717
 
 

 ##
 File path: sdks/java/extensions/sql/build.gradle
 ##
 @@ -45,23 +48,27 @@ dependencies {
   compile "org.apache.calcite:calcite-core:$calcite_version"
   compile "org.apache.calcite:calcite-linq4j:$calcite_version"
   compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
-  compile "com.alibaba:fastjson:1.2.12"
   compileOnly library.java.findbugs_jsr305
   compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow project(path: ":beam-sdks-java-extensions-join-library", 
configuration: "shadow")
   shadow library.java.slf4j_api
+  shadow library.java.slf4j_jdk14
   shadow library.java.commons_csv
+  shadow library.java.commons_lang3
+  shadow library.java.jackson_databind
   shadow library.java.joda_time
+  shadow "com.alibaba:fastjson:1.2.12"
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-kafka", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")
   provided library.java.kafka_clients
-  testCompile library.java.slf4j_jdk14
   testCompile library.java.junit
   testCompile library.java.hamcrest_core
   testCompile library.java.mockito_core
   testCompile library.java.quickcheck_core
+  shadowJarTest project(path: project.path, configuration: "shadow")
 
 Review comment:
   The tests run against the classes which aren't shaded (same as Maven before) 
since the default is `project.sourceSets.test.runtimeClasspath` as per [Gradle 
Test 
documentation](https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:classpath).
 Only the dependencies are correctly pulled in as shaded jars (an improvement 
over Maven).
   
   It might be better to attempt to run all unit tests using the shadowTest 
configuration by specifying `classpath = configurations.shadowTest` on the 
`test { }` closure in `build_rules.gradle`
   
   The test wouldn't need to become an `IT` then.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104793)
Time Spent: 1.5h  (was: 1h 20m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?focusedWorklogId=104792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104792
 ]

ASF GitHub Bot logged work on BEAM-4357:


Author: ASF GitHub Bot
Created on: 22/May/18 21:24
Start Date: 22/May/18 21:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5443: 
[BEAM-4357] [SQL] Fix shadow jar and test it
URL: https://github.com/apache/beam/pull/5443#discussion_r190057717
 
 

 ##
 File path: sdks/java/extensions/sql/build.gradle
 ##
 @@ -45,23 +48,27 @@ dependencies {
   compile "org.apache.calcite:calcite-core:$calcite_version"
   compile "org.apache.calcite:calcite-linq4j:$calcite_version"
   compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
-  compile "com.alibaba:fastjson:1.2.12"
   compileOnly library.java.findbugs_jsr305
   compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow project(path: ":beam-sdks-java-extensions-join-library", 
configuration: "shadow")
   shadow library.java.slf4j_api
+  shadow library.java.slf4j_jdk14
   shadow library.java.commons_csv
+  shadow library.java.commons_lang3
+  shadow library.java.jackson_databind
   shadow library.java.joda_time
+  shadow "com.alibaba:fastjson:1.2.12"
   shadow project(path: ":beam-runners-direct-java", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-kafka", configuration: "shadow")
   provided project(path: ":beam-sdks-java-io-google-cloud-platform", 
configuration: "shadow")
   provided library.java.kafka_clients
-  testCompile library.java.slf4j_jdk14
   testCompile library.java.junit
   testCompile library.java.hamcrest_core
   testCompile library.java.mockito_core
   testCompile library.java.quickcheck_core
+  shadowJarTest project(path: project.path, configuration: "shadow")
 
 Review comment:
   The tests run against the classes which aren't shaded (same as Maven before) 
since the default is `project.sourceSets.test.runtimeClasspath` as per [Gradle 
Test 
documentation](https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:classpath).
 Only the dependencies are correctly pulled in as shaded jars (an improvement 
over Maven).
   
   It might be better to attempt to run all unit tests using the shadowTest 
configuration by specifying `classpath = configurations.shadowTest` on the 
`test { }` closure in `build_rules.gradle`
   
   The test wouldn't need to become an IT then.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104792)
Time Spent: 1h 20m  (was: 1h 10m)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #321

[Apache Jenkins Server]

See 


Changes:

[timrobertson100] [BEAM-4341] Enforce ErrorProne analysis in 
google-cloud-platform IO

[tgroh] Add a Default RemoteEnvironment Implementation

[tgroh] Add a Window-only State Context

[tgroh] Add a BundleFactoryOutputReceiverFactory

[tgroh] Add a DirectJobBundleFactory

[tgroh] Fix Copyright Headers in sdks/java/fn-execution

[szewinho] [BEAM-4065] Basic performance tests analysis added.

[szewinho] Fixed typo in env name, added unit tests for create report logic, 
slack

[szewinho] Jenkins job will run tests before executing script. Extended testing 
of

[lcwik] [BEAM-4060] Add the pom.xml and pom.properties file to the artifact when

[github] [BEAM-1755] Add a directory with build-specific scripts to clear up the

--
[...truncated 18.16 MB...]
INFO: Adding Create123/Read(CreateSource) as step s10
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 22, 2018 9:16:35 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 22, 2018 9:16:35 PM 

[beam] branch master updated (a753d21 -> 30fa605)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a753d21  Merge pull request #5392: [BEAM-4267] JobBundleFactory that 
uses Docker-backed environments
 add 737a0c2  [BEAM-4348] Enforce ErrorProne analysis in kinesis IO
 add ae01977  [BEAM-4348] Removes double-checked locking to aid readability
 new 30fa605  Merge pull request #5448: [BEAM-4348] Enforce ErrorProne 
analysis in kinesis IO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/kinesis/build.gradle   |  4 +++-
 .../apache/beam/sdk/io/kinesis/ShardReadersPool.java|  2 ++
 .../apache/beam/sdk/io/kinesis/AmazonKinesisMock.java   | 13 -
 .../org/apache/beam/sdk/io/kinesis/KinesisIOIT.java |  3 ++-
 .../beam/sdk/io/kinesis/KinesisMockWriteTest.java   | 17 -
 .../beam/sdk/io/kinesis/KinesisRecordCoderTest.java |  3 ++-
 .../apache/beam/sdk/io/kinesis/KinesisServiceMock.java  |  8 ++--
 7 files changed, 31 insertions(+), 19 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Resolved] (BEAM-4348) Enforce ErrorProne analysis in the kinesis IO project

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4348.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Enforce ErrorProne analysis in the kinesis IO project
> -
>
> Key: BEAM-4348
> URL: https://issues.apache.org/jira/browse/BEAM-4348
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
> Fix For: 2.5.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kinesis}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kinesis:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kinesis}} project.
> # In {{sdks/java/io/kinesis/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4348) Enforce ErrorProne analysis in the kinesis IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4348?focusedWorklogId=104789=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104789
 ]

ASF GitHub Bot logged work on BEAM-4348:


Author: ASF GitHub Bot
Created on: 22/May/18 21:18
Start Date: 22/May/18 21:18
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #5448: [BEAM-4348] Enforce 
ErrorProne analysis in kinesis IO
URL: https://github.com/apache/beam/pull/5448
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/kinesis/build.gradle 
b/sdks/java/io/kinesis/build.gradle
index 05e49569185..4df04a26efa 100644
--- a/sdks/java/io/kinesis/build.gradle
+++ b/sdks/java/io/kinesis/build.gradle
@@ -17,7 +17,7 @@
  */
 
 apply from: project(":").file("build_rules.gradle")
-applyJavaNature()
+applyJavaNature(failOnWarning: true)
 provideIntegrationTestingDependencies()
 enableJavaPerformanceTesting()
 
@@ -33,6 +33,7 @@ def aws_version = "1.11.255"
 
 dependencies {
   compile library.java.guava
+  compileOnly library.java.findbugs_annotations
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow library.java.slf4j_api
   shadow library.java.joda_time
@@ -51,4 +52,5 @@ dependencies {
   testCompile library.java.hamcrest_core
   testCompile library.java.slf4j_jdk14
   testCompile "org.assertj:assertj-core:2.5.0"
+  testCompileOnly library.java.findbugs_annotations
 }
diff --git 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
index 67611a935a0..578c3516c76 100644
--- 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
+++ 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
@@ -106,6 +106,8 @@ void start() throws TransientKinesisException {
 }
   }
 
+  // Note: readLoop() will log any Throwable raised so opt to ignore the 
future result
+  @SuppressWarnings("FutureReturnValueIgnored")
   void startReadingShards(Iterable 
shardRecordsIterators) {
 for (final ShardRecordsIterator recordsIterator : shardRecordsIterators) {
   numberOfRecordsInAQueueByShard.put(recordsIterator.getShardId(), new 
AtomicInteger());
diff --git 
a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
 
b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
index f4906bdad86..7c5ca7da4db 100644
--- 
a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
+++ 
b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
@@ -80,8 +80,10 @@
 import com.amazonaws.services.kinesis.producer.IKinesisProducer;
 import com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration;
 import com.amazonaws.services.kinesis.waiters.AmazonKinesisWaiters;
+import com.google.common.base.Splitter;
 import java.io.Serializable;
 import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.stream.Collectors;
@@ -101,7 +103,7 @@
 private final String sequenceNumber;
 
 public TestData(KinesisRecord record) {
-  this(new String(record.getData().array()),
+  this(new String(record.getData().array(), StandardCharsets.UTF_8),
   record.getApproximateArrivalTimestamp(),
   record.getSequenceNumber());
 }
@@ -115,7 +117,7 @@ public TestData(String data, Instant arrivalTimestamp, 
String sequenceNumber) {
 public Record convertToRecord() {
   return new Record().
   withApproximateArrivalTimestamp(arrivalTimestamp.toDate()).
-  withData(ByteBuffer.wrap(data.getBytes())).
+  withData(ByteBuffer.wrap(data.getBytes(StandardCharsets.UTF_8))).
   withSequenceNumber(sequenceNumber).
   withPartitionKey("");
 }
@@ -176,9 +178,10 @@ public AmazonKinesisMock(List shardedData, 
int numberOfRecordsPerG
 
   @Override
   public GetRecordsResult getRecords(GetRecordsRequest getRecordsRequest) {
-String[] shardIteratorParts = 
getRecordsRequest.getShardIterator().split(":");
-int shardId = parseInt(shardIteratorParts[0]);
-int startingRecord = parseInt(shardIteratorParts[1]);
+List shardIteratorParts =
+Splitter.on(':').splitToList(getRecordsRequest.getShardIterator());
+int shardId = parseInt(shardIteratorParts.get(0));
+int startingRecord = parseInt(shardIteratorParts.get(1));
 List shardData = shardedData.get(shardId);
 
 int toIndex = 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=104784=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104784
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 22/May/18 21:05
Start Date: 22/May/18 21:05
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5273: 
[BEAM-3883] Adding Client to push artifacts to artifact staging service
URL: https://github.com/apache/beam/pull/5273#discussion_r190052740
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/portable_stager_test.py
 ##
 @@ -48,7 +48,10 @@ def tearDown(self):
 if self._remote_dir:
   shutil.rmtree(self._remote_dir)
 
-  def stage_files(self, files):
+  def _stage_files(self, files):
+"""
 
 Review comment:
   Made the change


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104784)
Time Spent: 18h 50m  (was: 18h 40m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4361) Document usage of HBase TableSnapshotInputFormat

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4361?focusedWorklogId=104783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104783
 ]

ASF GitHub Bot logged work on BEAM-4361:


Author: ASF GitHub Bot
Created on: 22/May/18 20:53
Start Date: 22/May/18 20:53
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #445: [BEAM-4361] 
Document usage of HBase TableSnapshotInputFormat
URL: https://github.com/apache/beam-site/pull/445#issuecomment-391136836
 
 
   Broken build is due to a dead external link which is not related to this PR. 
Another PR addresses that issue. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104783)
Time Spent: 0.5h  (was: 20m)

> Document usage of HBase TableSnapshotInputFormat 
> -
>
> Key: BEAM-4361
> URL: https://issues.apache.org/jira/browse/BEAM-4361
> Project: Beam
>  Issue Type: Task
>  Components: website
>Affects Versions: 2.4.0
>Reporter: Tim Robertson
>Assignee: Tim Robertson
>Priority: Trivial
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add a paragraph demonstrating the usage of {{TableSnapshotInputFormat}} as a 
> mechanism for doing efficient full scans over HBase to 
> https://beam.apache.org/documentation/io/built-in/hadoop/
> Typically in MR / Spark this yields up to 3-4x improvement over hitting 
> region servers directly and keeps load (GC etc) from those services.
> I have it tested and an example ready.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4361) Document usage of HBase TableSnapshotInputFormat

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4361?focusedWorklogId=104781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104781
 ]

ASF GitHub Bot logged work on BEAM-4361:


Author: ASF GitHub Bot
Created on: 22/May/18 20:49
Start Date: 22/May/18 20:49
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #445: [BEAM-4361] 
Document usage of HBase TableSnapshotInputFormat
URL: https://github.com/apache/beam-site/pull/445#issuecomment-391135863
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104781)
Time Spent: 20m  (was: 10m)

> Document usage of HBase TableSnapshotInputFormat 
> -
>
> Key: BEAM-4361
> URL: https://issues.apache.org/jira/browse/BEAM-4361
> Project: Beam
>  Issue Type: Task
>  Components: website
>Affects Versions: 2.4.0
>Reporter: Tim Robertson
>Assignee: Tim Robertson
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add a paragraph demonstrating the usage of {{TableSnapshotInputFormat}} as a 
> mechanism for doing efficient full scans over HBase to 
> https://beam.apache.org/documentation/io/built-in/hadoop/
> Typically in MR / Spark this yields up to 3-4x improvement over hitting 
> region servers directly and keeps load (GC etc) from those services.
> I have it tested and an example ready.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=104779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104779
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 22/May/18 20:46
Start Date: 22/May/18 20:46
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5273: 
[BEAM-3883] Adding Client to push artifacts to artifact staging service
URL: https://github.com/apache/beam/pull/5273#discussion_r190047124
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/portable_stager_test.py
 ##
 @@ -48,7 +48,10 @@ def tearDown(self):
 if self._remote_dir:
   shutil.rmtree(self._remote_dir)
 
-  def stage_files(self, files):
+  def _stage_files(self, files):
+"""
 
 Review comment:
   The reason I asked for docstring is that it would be useful to describe the 
structure of `files` since it is not obvious without reading the method. How 
about:
   
   ```
   Utility method to stage files.
   
   Args: 
   
 files: a list of tuples of the form [(local_name, remote_name),...] 
describing the name of the artifacts in local temp folder and desired name in 
staging location. 
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104779)
Time Spent: 18h 40m  (was: 18.5h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 18h 40m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #513

[Apache Jenkins Server]

See 


Changes:

[sidhom] Require explicit IdGenerator for all DockerEnvironmentFactory

[sidhom] [BEAM-4148] Add pluggable UrlFactory to ServerFactory

[sidhom] [BEAM-4267] Docker-based JobBundleFactory

--
[...truncated 17.87 MB...]
May 22, 2018 8:38:17 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Write 
mutations to Spanner as step s40
May 22, 2018 8:38:17 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0522203808-2037a755/output/results/staging/
May 22, 2018 8:38:17 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <115879 bytes, hash dPkwCh_r6GlPiQccdWs6RQ> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0522203808-2037a755/output/results/staging/pipeline-dPkwCh_r6GlPiQccdWs6RQ.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 22, 2018 8:38:18 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-22_13_38_17-14123995809600494030?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-22_13_38_17-14123995809600494030

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 22, 2018 8:38:18 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-22_13_38_17-14123995809600494030
May 22, 2018 8:38:18 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-22_13_38_17-14123995809600494030 with 0 
expected assertions.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:17.722Z: Autoscaling is enabled for job 
2018-05-22_13_38_17-14123995809600494030. The number of workers will be between 
1 and 1000.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:17.759Z: Autoscaling was automatically enabled for 
job 2018-05-22_13_38_17-14123995809600494030.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:20.753Z: Checking required Cloud APIs are enabled.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:20.880Z: Checking permissions granted to controller 
Service Account.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:24.726Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.069Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.360Z: Expanding GroupByKey operations into 
optimizable parts.
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.397Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.688Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.744Z: Elided trivial flatten 
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.795Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 22, 2018 8:38:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-22T20:38:25.838Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud 

[jira] [Work logged] (BEAM-4348) Enforce ErrorProne analysis in the kinesis IO project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4348?focusedWorklogId=104778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-104778
 ]

ASF GitHub Bot logged work on BEAM-4348:


Author: ASF GitHub Bot
Created on: 22/May/18 20:29
Start Date: 22/May/18 20:29
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #5448: [BEAM-4348] 
Enforce ErrorProne analysis in kinesis IO
URL: https://github.com/apache/beam/pull/5448#issuecomment-391129911
 
 
   PTAL @iemejia 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 104778)
Time Spent: 40m  (was: 0.5h)

> Enforce ErrorProne analysis in the kinesis IO project
> -
>
> Key: BEAM-4348
> URL: https://issues.apache.org/jira/browse/BEAM-4348
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kinesis}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kinesis:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kinesis}} project.
> # In {{sdks/java/io/kinesis/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)