[jira] [Commented] (BEAM-5172) org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591633#comment-16591633
 ] 

Etienne Chauchot commented on BEAM-5172:


There are 2 types of failure:
- timeout in the connection to embedded ES
- assert of number of docs in index fail: the index contains more docs than 
inserted. Maybe the index drop was not finished before running the test

> org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky
> -
>
> Key: BEAM-5172
> URL: https://issues.apache.org/jira/browse/BEAM-5172
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-elasticsearch, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Etienne Chauchot
>Priority: Major
>
> In a recent PostCommit builld, 
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1290/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testRead/
>  failed with:
> Error Message
> java.lang.AssertionError: Count/Flatten.PCollections.out: 
> Expected: <400L>
>  but: was <470L>
> Stacktrace
> java.lang.AssertionError: Count/Flatten.PCollections.out: 
> Expected: <400L>
>  but: was <470L>
>   at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
>   at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:413)
>   at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:404)
>   at 
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.testRead(ElasticsearchIOTestCommon.java:124)
>   at 
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testRead(ElasticsearchIOTest.java:125)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispa

[jira] [Work logged] (BEAM-3026) Improve retrying in ElasticSearch client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3026?focusedWorklogId=137800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137800
 ]

ASF GitHub Bot logged work on BEAM-3026:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:20
Start Date: 24/Aug/18 13:20
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #6146: [BEAM-3026] Adding 
retrying behavior on ElasticSearchIO
URL: https://github.com/apache/beam/pull/6146#issuecomment-415756571
 
 
   build was ok before squash and now fails with `The 
:beam-runners-spark:compileTestJava task failed.` Build is flaky and the error 
is unrelated to this PR. Merging


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137800)
Time Spent: 15h 10m  (was: 15h)

> Improve retrying in ElasticSearch client
> 
>
> Key: BEAM-3026
> URL: https://issues.apache.org/jira/browse/BEAM-3026
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-elasticsearch
>Reporter: Tim Robertson
>Assignee: Ravi Pathak
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 15h 10m
>  Remaining Estimate: 0h
>
> Currently an overloaded ES server will result in clients failing fast.
> I suggest implementing backoff pauses.  Perhaps something like this:
> {code}
> ElasticsearchIO.ConnectionConfiguration conn = 
> ElasticsearchIO.ConnectionConfiguration
>   .create(new String[]{"http://...:9200"}, "test", "test")
>   .retryWithWaitStrategy(WaitStrategies.exponentialBackoff(1000, 
> TimeUnit.MILLISECONDS)
>   .retryWithStopStrategy(StopStrategies.stopAfterAttempt(10)
> );
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137807
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212589676
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingExecutableStageContext.java
 ##
 @@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.control.DockerJobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+
+/** Implementation of a {@link FlinkExecutableStageContext} for streaming. */
+public class FlinkStreamingExecutableStageContext
 
 Review comment:
   Does this class add any additional functionality to the 
`FlinkBatchExecutableStageContext`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137807)
Time Spent: 5.5h  (was: 5h 20m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137801
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212317984
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -149,6 +149,7 @@ public static FlinkBatchPortablePipelineTranslator 
createTranslator() {
 translatorMap.put(
 PTransformTranslation.RESHUFFLE_URN,
 FlinkBatchPortablePipelineTranslator::translateReshuffle);
+// TODO: this does not seem required
 translatorMap.put(
 PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
 FlinkBatchPortablePipelineTranslator::translateView);
 
 Review comment:
   Not sure if that always work, e.g. when the input stream of the view is not 
already added by another transform?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137801)
Time Spent: 4h 40m  (was: 4.5h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137806
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212337222
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java
 ##
 @@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.SideInputReference;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import 
org.apache.beam.runners.fnexecution.state.StateRequestHandlers.SideInputHandler;
+import 
org.apache.beam.runners.fnexecution.state.StateRequestHandlers.SideInputHandlerFactory;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.transforms.Materializations;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
+
+/**
+ * {@link StateRequestHandler} that uses {@link 
org.apache.beam.runners.core.SideInputHandler} to
+ * access the Flink broadcast state that represents side inputs.
+ */
+public class FlinkStreamingSideInputHandlerFactory implements 
SideInputHandlerFactory {
+
+  // Map from side input id to global PCollection id.
+  private final Map> sideInputToCollection;
+  private final org.apache.beam.runners.core.SideInputHandler runnerHandler;
+
+  /**
+   * Creates a new state handler for the given stage. Note that this requires 
a traversal of the
+   * stage itself, so this should only be called once per stage rather than 
once per bundle.
+   */
+  public static FlinkStreamingSideInputHandlerFactory forStage(
+  ExecutableStage stage,
+  Map> viewMapping,
+  org.apache.beam.runners.core.SideInputHandler runnerHandler) {
+ImmutableMap.Builder> sideInputBuilder = 
ImmutableMap.builder();
+for (SideInputReference sideInput : stage.getSideInputs()) {
+  SideInputId sideInputId =
+  SideInputId.newBuilder()
+  .setTransformId(sideInput.transform().getId())
+  .setLocalName(sideInput.localName())
+  .build();
+  sideInputBuilder.put(
+  sideInputId,
+  checkNotNull(
+  viewMapping.get(sideInputId),
+  "No side input for %s/%s",
+  sideInputId.getTransformId(),
+  sideInputId.getLocalName()));
+}
+
+FlinkStreamingSideInputHandlerFactory factory =
+new FlinkStreamingSideInputHandlerFactory(sideInputBuilder.build(), 
runnerHandler);
+return factory;
+  }
+
+  private FlinkStreamingSideInputHandlerFactory(
+  Map> sideInputToCollection,
+  org.apache.beam.runners.core.SideInputHandler runnerHandler) {
+this.sideInputToCollection = sideInputToCollection;
+this.runnerHandler = runnerHandler;
+  }
+
+  @Override
+  public  SideInputHandler forSideInput(
+  String transformId,
+  String sideInputId,
+  RunnerApi.FunctionSpec accessPattern,
+  Coder elementCoder,
+

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137802
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212581542
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -520,6 +567,158 @@ private void translateImpulse(
 }
   }
 
+  private static LinkedHashMap>
+  getSideInputIdToPCollectionViewMap(
+  RunnerApi.ExecutableStagePayload stagePayload, RunnerApi.Components 
components) {
+
+RehydratedComponents rehydratedComponents = 
RehydratedComponents.forComponents(components);
+
+LinkedHashMap> sideInputs =
+new LinkedHashMap<>();
+for (RunnerApi.ExecutableStagePayload.SideInputId sideInputId :
+stagePayload.getSideInputsList()) {
+
+  // TODO: local name is unique as long as only one transform with side 
input can be within a stage
+  String sideInputTag = sideInputId.getLocalName();
+  // for PCollectionView compatibility, not used to transform 
materialization
+  ViewFn>, ?> viewFn =
+  (ViewFn) new 
PCollectionViews.MultimapViewFn>, Void>();
 
 Review comment:
   This can be moved below to the constructor or even outside the loop.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137802)
Time Spent: 4h 50m  (was: 4h 40m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6146 from aalbatross/feature/es-retry

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5aaf584c9b2dd773d1112f59b66d1b9fc0e64d3f
Merge: 0299676 90ca0ce
Author: Etienne Chauchot 
AuthorDate: Fri Aug 24 15:22:16 2018 +0200

Merge pull request #6146 from aalbatross/feature/es-retry

Merge pull request #6146: [BEAM-3026] Adding retrying behavior on 
ElasticSearchIO

 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |  12 ++
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |  14 ++
 .../elasticsearch/ElasticsearchIOTestCommon.java   |  61 ++
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 204 -
 4 files changed, 287 insertions(+), 4 deletions(-)

[beam] branch master updated (0299676 -> 5aaf584)

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0299676  Merge pull request #6272: Enable streaming flag in 
portableWordCount.
 add 90ca0ce  [BEAM-3026] Adding retrying behavior on ESIO for error http 
429 (Too Many Requests)
 new 5aaf584  Merge pull request #6146 from aalbatross/feature/es-retry

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |  12 ++
 .../sdk/io/elasticsearch/ElasticsearchIOTest.java  |  14 ++
 .../elasticsearch/ElasticsearchIOTestCommon.java   |  61 ++
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 204 -
 4 files changed, 287 insertions(+), 4 deletions(-)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137805
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212340352
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -108,27 +119,45 @@ public ExecutableStageDoFnOperator(
 this.jobInfo = jobInfo;
 this.contextFactory = contextFactory;
 this.outputMap = outputMap;
+this.sideInputIds = sideInputIds;
   }
 
   @Override
   public void open() throws Exception {
 super.open();
 
-ExecutableStage executableStage = ExecutableStage.fromPayload(payload);
+executableStage = ExecutableStage.fromPayload(payload);
 // TODO: Wire this into the distributed cache and make it pluggable.
 // TODO: Do we really want this layer of indirection when accessing the 
stage bundle factory?
 // It's a little strange because this operator is responsible for the 
lifetime of the stage
 // bundle "factory" (manager?) but not the job or Flink bundle factories. 
How do we make
 // ownership of the higher level "factories" explicit? Do we care?
 stageContext = contextFactory.get(jobInfo);
-// NOTE: It's safe to reuse the state handler between partitions because 
each partition uses the
-// same backing runtime context and broadcast variables. We use checkState 
below to catch errors
-// in backward-incompatible Flink changes.
-stateRequestHandler = stageContext.getStateRequestHandler(executableStage, 
getRuntimeContext());
+
+stateRequestHandler = getStateRequestHandler(executableStage);
 stageBundleFactory = stageContext.getStageBundleFactory(executableStage);
 progressHandler = BundleProgressHandler.unsupported();
   }
 
+  private StateRequestHandler getStateRequestHandler(ExecutableStage 
executableStage) {
+
+if (!executableStage.getSideInputs().isEmpty()) {
 
 Review comment:
   Not important but I prefer: `if (executableStage.getSideInputs() > 0) {` or 
switch around the if/else blocks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137805)
Time Spent: 5h 20m  (was: 5h 10m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137808
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212585550
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -520,6 +567,158 @@ private void translateImpulse(
 }
   }
 
+  private static LinkedHashMap>
+  getSideInputIdToPCollectionViewMap(
+  RunnerApi.ExecutableStagePayload stagePayload, RunnerApi.Components 
components) {
+
+RehydratedComponents rehydratedComponents = 
RehydratedComponents.forComponents(components);
+
+LinkedHashMap> sideInputs =
+new LinkedHashMap<>();
+for (RunnerApi.ExecutableStagePayload.SideInputId sideInputId :
+stagePayload.getSideInputsList()) {
+
+  // TODO: local name is unique as long as only one transform with side 
input can be within a stage
+  String sideInputTag = sideInputId.getLocalName();
+  // for PCollectionView compatibility, not used to transform 
materialization
+  ViewFn>, ?> viewFn =
+  (ViewFn) new 
PCollectionViews.MultimapViewFn>, Void>();
+
+  String collectionId =
+  components
+  .getTransformsOrThrow(sideInputId.getTransformId())
+  .getInputsOrThrow(sideInputId.getLocalName());
+  RunnerApi.WindowingStrategy windowingStrategyProto =
+  components.getWindowingStrategiesOrThrow(
+  
components.getPcollectionsOrThrow(collectionId).getWindowingStrategyId());
+
+  final WindowingStrategy windowingStrategy;
+  try {
+windowingStrategy =
+WindowingStrategyTranslation.fromProto(windowingStrategyProto, 
rehydratedComponents);
+  } catch (InvalidProtocolBufferException e) {
+throw new IllegalStateException(
+String.format(
+"Unable to hydrate side input windowing strategy %s.", 
windowingStrategyProto),
+e);
+  }
+
+  Coder> coder = instantiateCoder(collectionId, 
components);
+  // side input materialization via GBK (T -> Iterable)
+  WindowedValueCoder wvCoder = (WindowedValueCoder) coder;
+  coder = 
wvCoder.withValueCoder(IterableCoder.of(wvCoder.getValueCoder()));
+
+  sideInputs.put(
+  sideInputId,
+  new RunnerPCollectionView<>(
+  null,
+  new TupleTag<>(sideInputTag),
+  viewFn,
+  // TODO: support custom mapping fn
+  windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
+  windowingStrategy,
+  coder));
+}
+return sideInputs;
+  }
+
+  private Tuple2>, DataStream> 
transformSideInputs(
+  RunnerApi.ExecutableStagePayload stagePayload,
+  RunnerApi.Components components,
+  StreamingTranslationContext context) {
+
+LinkedHashMap> sideInputs =
+getSideInputIdToPCollectionViewMap(stagePayload, components);
+
+Map, Integer> tagToIntMapping = new HashMap<>();
+Map> intToViewMapping = new HashMap<>();
+List>> kvCoders = new ArrayList<>();
+List> viewCoders = new ArrayList<>();
+
+int count = 0;
+for (Map.Entry> sideInput :
+sideInputs.entrySet()) {
+  TupleTag tag = sideInput.getValue().getTagInternal();
+  intToViewMapping.put(count, sideInput.getValue());
+  tagToIntMapping.put(tag, count);
+  count++;
+  String collectionId =
+  components
+  .getTransformsOrThrow(sideInput.getKey().getTransformId())
+  .getInputsOrThrow(sideInput.getKey().getLocalName());
+  DataStream sideInputStream = 
context.getDataStreamOrThrow(collectionId);
+  TypeInformation tpe = sideInputStream.getType();
+  if (!(tpe instanceof CoderTypeInformation)) {
+throw new IllegalStateException("Input Stream TypeInformation is no 
CoderTypeInformation.");
+  }
+
+  WindowedValueCoder coder =
+  (WindowedValueCoder) ((CoderTypeInformation) tpe).getCoder();
+  Coder> kvCoder = KvCoder.of(VoidCoder.of(), 
coder.getValueCoder());
+  kvCoders.add(coder.withValueCoder(kvCoder));
+  // coder for materialized view matching GBK below
+  WindowedValueCoder>> viewCoder =
+  coder.withValueCoder(KvCoder.of(VoidCoder.of(), 
IterableCoder.of(coder.getValueCoder(;
+  viewCoders.add(viewCoder);
+}
+
+// second pass, now that we gathered the input coders
+UnionCoder unionCoder = UnionCoder.of(viewCoders);
+
+CoderTypeInformation unionTypeIn

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137804
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212574656
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 ##
 @@ -148,30 +147,45 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
 Review comment:
   Shouldn't this go into the `getIterable` function?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137804)
Time Spent: 5h 10m  (was: 5h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137803
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:21
Start Date: 24/Aug/18 13:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212575795
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 ##
 @@ -148,30 +147,45 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
+// TODO: Add support for choosing which representation is contained based 
upon the
+// side input materialization. We currently can assume that we always have 
a multimap
+// materialization as that is the only supported type within the Java SDK.
 ViewFn viewFn = (ViewFn) 
view.getViewFn();
 
 Review comment:
   Could we add an `instanceof` check and check whether this is 
`MultimapViewFn`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137803)
Time Spent: 5h  (was: 4h 50m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3026) Improve retrying in ElasticSearch client

[Etienne Chauchot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Etienne Chauchot resolved BEAM-3026.

Resolution: Fixed

> Improve retrying in ElasticSearch client
> 
>
> Key: BEAM-3026
> URL: https://issues.apache.org/jira/browse/BEAM-3026
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-elasticsearch
>Reporter: Tim Robertson
>Assignee: Ravi Pathak
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 15h 20m
>  Remaining Estimate: 0h
>
> Currently an overloaded ES server will result in clients failing fast.
> I suggest implementing backoff pauses.  Perhaps something like this:
> {code}
> ElasticsearchIO.ConnectionConfiguration conn = 
> ElasticsearchIO.ConnectionConfiguration
>   .create(new String[]{"http://...:9200"}, "test", "test")
>   .retryWithWaitStrategy(WaitStrategies.exponentialBackoff(1000, 
> TimeUnit.MILLISECONDS)
>   .retryWithStopStrategy(StopStrategies.stopAfterAttempt(10)
> );
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3026) Improve retrying in ElasticSearch client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3026?focusedWorklogId=137809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137809
 ]

ASF GitHub Bot logged work on BEAM-3026:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:22
Start Date: 24/Aug/18 13:22
Worklog Time Spent: 10m 
  Work Description: echauchot closed pull request #6146: [BEAM-3026] Adding 
retrying behavior on ElasticSearchIO
URL: https://github.com/apache/beam/pull/6146
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
index 1e3e7b62dfb..862ba5b4292 100644
--- 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
+++ 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
@@ -220,4 +220,16 @@ public void testReadWithMetadata() throws Exception {
 elasticsearchIOTestCommon.setPipeline(pipeline);
 elasticsearchIOTestCommon.testReadWithMetadata();
   }
+
+  @Test
+  public void testDefaultRetryPredicate() throws IOException {
+elasticsearchIOTestCommon.testDefaultRetryPredicate(restClient);
+  }
+
+  @Test
+  public void testWriteRetry() throws Throwable {
+elasticsearchIOTestCommon.setExpectedException(expectedException);
+elasticsearchIOTestCommon.setPipeline(pipeline);
+elasticsearchIOTestCommon.testWriteRetry();
+  }
 }
diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
index c28b1906f84..d2791c76d1f 100644
--- 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
+++ 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
@@ -26,8 +26,10 @@
 import static 
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.NUM_DOCS_UTESTS;
 import static org.apache.beam.sdk.testing.SourceTestUtils.readFromSource;
 import static org.hamcrest.Matchers.lessThan;
+import static org.junit.Assert.assertEquals;
 
 import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
+import java.io.IOException;
 import java.io.Serializable;
 import java.net.InetSocketAddress;
 import java.util.ArrayList;
@@ -222,4 +224,16 @@ public void testReadWithMetadata() throws Exception {
 elasticsearchIOTestCommon.setPipeline(pipeline);
 elasticsearchIOTestCommon.testReadWithMetadata();
   }
+
+  @Test
+  public void testDefaultRetryPredicate() throws IOException {
+elasticsearchIOTestCommon.testDefaultRetryPredicate(getRestClient());
+  }
+
+  @Test
+  public void testWriteRetry() throws Throwable {
+elasticsearchIOTestCommon.setExpectedException(expectedException);
+elasticsearchIOTestCommon.setPipeline(pipeline);
+elasticsearchIOTestCommon.testWriteRetry();
+  }
 }
diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
index dedc49f0ea3..57b450d8c9e 100644
--- 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
+++ 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
@@ -25,11 +25,14 @@
 import static 
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BoundedElasticsearchSource;
 import static 
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration;
 import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Read;
+import static 
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DEFAULT_RETRY_PREDICATE;
 import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.core.Is.isA;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
 import static org.ju

[jira] [Comment Edited] (BEAM-5172) org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591633#comment-16591633
 ] 

Etienne Chauchot edited comment on BEAM-5172 at 8/24/18 1:25 PM:
-

There are 2 types of failure:
- timeout in the connection to embedded ES => overload of jenkins ? Need to 
block until embedded ES is ready?
- assert of number of docs in index fail: the index contains more docs than 
inserted. Maybe the index drop was not finished before running the test


was (Author: echauchot):
There are 2 types of failure:
- timeout in the connection to embedded ES
- assert of number of docs in index fail: the index contains more docs than 
inserted. Maybe the index drop was not finished before running the test

> org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky
> -
>
> Key: BEAM-5172
> URL: https://issues.apache.org/jira/browse/BEAM-5172
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-elasticsearch, test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Etienne Chauchot
>Priority: Major
>
> In a recent PostCommit builld, 
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1290/testReport/junit/org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest/testRead/
>  failed with:
> Error Message
> java.lang.AssertionError: Count/Flatten.PCollections.out: 
> Expected: <400L>
>  but: was <470L>
> Stacktrace
> java.lang.AssertionError: Count/Flatten.PCollections.out: 
> Expected: <400L>
>  but: was <470L>
>   at 
> org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168)
>   at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:413)
>   at org.apache.beam.sdk.testing.PAssert.thatSingleton(PAssert.java:404)
>   at 
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.testRead(ElasticsearchIOTestCommon.java:124)
>   at 
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testRead(ElasticsearchIOTest.java:125)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137824&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137824
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 13:58
Start Date: 24/Aug/18 13:58
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212636822
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,14 +204,10 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
 
 Review comment:
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137824)
Time Spent: 5h 20m  (was: 5h 10m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1340

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137858
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:51
Start Date: 24/Aug/18 15:51
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212673728
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount 
--job_endpoint=localhost:8099
 
 Review comment:
 Doesn't this require a project property?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137858)
Time Spent: 5.5h  (was: 5h 20m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137859
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:54
Start Date: 24/Aug/18 15:54
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212674635
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingExecutableStageContext.java
 ##
 @@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.control.DockerJobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+
+/** Implementation of a {@link FlinkExecutableStageContext} for streaming. */
+public class FlinkStreamingExecutableStageContext
 
 Review comment:
   None, removed it. (It was necessary before when it was entangled with the 
state request handler.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137859)
Time Spent: 5h 40m  (was: 5.5h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137860
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:55
Start Date: 24/Aug/18 15:55
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212674697
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -520,6 +567,158 @@ private void translateImpulse(
 }
   }
 
+  private static LinkedHashMap>
+  getSideInputIdToPCollectionViewMap(
+  RunnerApi.ExecutableStagePayload stagePayload, RunnerApi.Components 
components) {
+
+RehydratedComponents rehydratedComponents = 
RehydratedComponents.forComponents(components);
+
+LinkedHashMap> sideInputs =
+new LinkedHashMap<>();
+for (RunnerApi.ExecutableStagePayload.SideInputId sideInputId :
+stagePayload.getSideInputsList()) {
+
+  // TODO: local name is unique as long as only one transform with side 
input can be within a stage
+  String sideInputTag = sideInputId.getLocalName();
+  // for PCollectionView compatibility, not used to transform 
materialization
+  ViewFn>, ?> viewFn =
+  (ViewFn) new 
PCollectionViews.MultimapViewFn>, Void>();
+
+  String collectionId =
+  components
+  .getTransformsOrThrow(sideInputId.getTransformId())
+  .getInputsOrThrow(sideInputId.getLocalName());
+  RunnerApi.WindowingStrategy windowingStrategyProto =
+  components.getWindowingStrategiesOrThrow(
+  
components.getPcollectionsOrThrow(collectionId).getWindowingStrategyId());
+
+  final WindowingStrategy windowingStrategy;
+  try {
+windowingStrategy =
+WindowingStrategyTranslation.fromProto(windowingStrategyProto, 
rehydratedComponents);
+  } catch (InvalidProtocolBufferException e) {
+throw new IllegalStateException(
+String.format(
+"Unable to hydrate side input windowing strategy %s.", 
windowingStrategyProto),
+e);
+  }
+
+  Coder> coder = instantiateCoder(collectionId, 
components);
+  // side input materialization via GBK (T -> Iterable)
+  WindowedValueCoder wvCoder = (WindowedValueCoder) coder;
+  coder = 
wvCoder.withValueCoder(IterableCoder.of(wvCoder.getValueCoder()));
+
+  sideInputs.put(
+  sideInputId,
+  new RunnerPCollectionView<>(
+  null,
+  new TupleTag<>(sideInputTag),
+  viewFn,
+  // TODO: support custom mapping fn
+  windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
+  windowingStrategy,
+  coder));
+}
+return sideInputs;
+  }
+
+  private Tuple2>, DataStream> 
transformSideInputs(
+  RunnerApi.ExecutableStagePayload stagePayload,
+  RunnerApi.Components components,
+  StreamingTranslationContext context) {
+
+LinkedHashMap> sideInputs =
+getSideInputIdToPCollectionViewMap(stagePayload, components);
+
+Map, Integer> tagToIntMapping = new HashMap<>();
+Map> intToViewMapping = new HashMap<>();
+List>> kvCoders = new ArrayList<>();
+List> viewCoders = new ArrayList<>();
+
+int count = 0;
+for (Map.Entry> sideInput :
+sideInputs.entrySet()) {
+  TupleTag tag = sideInput.getValue().getTagInternal();
+  intToViewMapping.put(count, sideInput.getValue());
+  tagToIntMapping.put(tag, count);
+  count++;
+  String collectionId =
+  components
+  .getTransformsOrThrow(sideInput.getKey().getTransformId())
+  .getInputsOrThrow(sideInput.getKey().getLocalName());
+  DataStream sideInputStream = 
context.getDataStreamOrThrow(collectionId);
+  TypeInformation tpe = sideInputStream.getType();
+  if (!(tpe instanceof CoderTypeInformation)) {
+throw new IllegalStateException("Input Stream TypeInformation is no 
CoderTypeInformation.");
+  }
+
+  WindowedValueCoder coder =
+  (WindowedValueCoder) ((CoderTypeInformation) tpe).getCoder();
+  Coder> kvCoder = KvCoder.of(VoidCoder.of(), 
coder.getValueCoder());
+  kvCoders.add(coder.withValueCoder(kvCoder));
+  // coder for materialized view matching GBK below
+  WindowedValueCoder>> viewCoder =
+  coder.withValueCoder(KvCoder.of(VoidCoder.of(), 
IterableCoder.of(coder.getValueCoder(;
+  viewCoders.add(viewCoder);
+}
+
+// second pass, now that we gathered the input coders
+UnionCoder unionCoder = UnionCoder.of(viewCoders);
+
+CoderTypeInformation unionTyp

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137861
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:55
Start Date: 24/Aug/18 15:55
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212674746
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -520,6 +567,158 @@ private void translateImpulse(
 }
   }
 
+  private static LinkedHashMap>
+  getSideInputIdToPCollectionViewMap(
+  RunnerApi.ExecutableStagePayload stagePayload, RunnerApi.Components 
components) {
+
+RehydratedComponents rehydratedComponents = 
RehydratedComponents.forComponents(components);
+
+LinkedHashMap> sideInputs =
+new LinkedHashMap<>();
+for (RunnerApi.ExecutableStagePayload.SideInputId sideInputId :
+stagePayload.getSideInputsList()) {
+
+  // TODO: local name is unique as long as only one transform with side 
input can be within a stage
+  String sideInputTag = sideInputId.getLocalName();
+  // for PCollectionView compatibility, not used to transform 
materialization
+  ViewFn>, ?> viewFn =
+  (ViewFn) new 
PCollectionViews.MultimapViewFn>, Void>();
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137861)
Time Spent: 6h  (was: 5h 50m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137862
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:56
Start Date: 24/Aug/18 15:56
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212675152
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 ##
 @@ -148,30 +147,45 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
+// TODO: Add support for choosing which representation is contained based 
upon the
+// side input materialization. We currently can assume that we always have 
a multimap
+// materialization as that is the only supported type within the Java SDK.
 ViewFn viewFn = (ViewFn) 
view.getViewFn();
 
 Review comment:
   Left as is, this was just moved around. It is an SDK specific assumption, 
portable runner does not use this code path.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137862)
Time Spent: 6h 10m  (was: 6h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (5aaf584 -> 2b777a1)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5aaf584  Merge pull request #6146 from aalbatross/feature/es-retry
 add b7af903  [BEAM-5184] Side inputs aggregating multimap changed from 
`HashMultimap` to `ArrayListMultimap`, which allows for duplicated key-value 
items. Tests in `ViewTest` modified to exercise duplicate key-value items 
scenario.
 add 0de6a0e  Fix-up BatchViewOverrides.java to use list based multimap.
 add bd8e96f  Merge pull request #25 from lukecwik/pr6257
 new 2b777a1  [BEAM-5184] Multimap side inputs with duplicate keys and 
values are being lost

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/dataflow/BatchViewOverrides.java  |  6 +--
 .../runners/dataflow/BatchViewOverridesTest.java   |  8 +++-
 .../apache/beam/sdk/values/PCollectionViews.java   |  4 +-
 .../org/apache/beam/sdk/transforms/ViewTest.java   | 43 +-
 4 files changed, 46 insertions(+), 15 deletions(-)

[beam] 01/01: [BEAM-5184] Multimap side inputs with duplicate keys and values are being lost

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2b777a109e8fdb2dcf81004fffcada3d7375f9ec
Merge: 5aaf584 bd8e96f
Author: Lukasz Cwik 
AuthorDate: Fri Aug 24 08:57:33 2018 -0700

[BEAM-5184] Multimap side inputs with duplicate keys and values are being 
lost

 .../beam/runners/dataflow/BatchViewOverrides.java  |  6 +--
 .../runners/dataflow/BatchViewOverridesTest.java   |  8 +++-
 .../apache/beam/sdk/values/PCollectionViews.java   |  4 +-
 .../org/apache/beam/sdk/transforms/ViewTest.java   | 43 +-
 4 files changed, 46 insertions(+), 15 deletions(-)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137866&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137866
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:57
Start Date: 24/Aug/18 15:57
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r21267
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -108,27 +119,45 @@ public ExecutableStageDoFnOperator(
 this.jobInfo = jobInfo;
 this.contextFactory = contextFactory;
 this.outputMap = outputMap;
+this.sideInputIds = sideInputIds;
   }
 
   @Override
   public void open() throws Exception {
 super.open();
 
-ExecutableStage executableStage = ExecutableStage.fromPayload(payload);
+executableStage = ExecutableStage.fromPayload(payload);
 // TODO: Wire this into the distributed cache and make it pluggable.
 // TODO: Do we really want this layer of indirection when accessing the 
stage bundle factory?
 // It's a little strange because this operator is responsible for the 
lifetime of the stage
 // bundle "factory" (manager?) but not the job or Flink bundle factories. 
How do we make
 // ownership of the higher level "factories" explicit? Do we care?
 stageContext = contextFactory.get(jobInfo);
-// NOTE: It's safe to reuse the state handler between partitions because 
each partition uses the
-// same backing runtime context and broadcast variables. We use checkState 
below to catch errors
-// in backward-incompatible Flink changes.
-stateRequestHandler = stageContext.getStateRequestHandler(executableStage, 
getRuntimeContext());
+
+stateRequestHandler = getStateRequestHandler(executableStage);
 stageBundleFactory = stageContext.getStageBundleFactory(executableStage);
 progressHandler = BundleProgressHandler.unsupported();
   }
 
+  private StateRequestHandler getStateRequestHandler(ExecutableStage 
executableStage) {
+
+if (!executableStage.getSideInputs().isEmpty()) {
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137866)
Time Spent: 6h 40m  (was: 6.5h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=137864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137864
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:57
Start Date: 24/Aug/18 15:57
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6257: [BEAM-5184] 
Multimap side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
index 145a6a23e1b..74ffac74378 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
@@ -22,8 +22,8 @@
 import com.google.common.base.Function;
 import com.google.common.base.MoreObjects;
 import com.google.common.base.Optional;
+import com.google.common.collect.ArrayListMultimap;
 import com.google.common.collect.ForwardingMap;
-import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Maps;
@@ -637,7 +637,7 @@ public void processElement(ProcessContext c) throws 
Exception {
   public void processElement(ProcessContext c) throws Exception {
 Optional previousWindowStructuralValue = Optional.absent();
 Optional previousWindow = Optional.absent();
-Multimap> multimap = HashMultimap.create();
+Multimap> multimap = ArrayListMultimap.create();
 for (KV>> kv : c.element().getValue()) {
   Object currentWindowStructuralValue = 
windowCoder.structuralValue(kv.getKey());
   if (previousWindowStructuralValue.isPresent()
@@ -652,7 +652,7 @@ public void processElement(ProcessContext c) throws 
Exception {
 valueInEmptyWindows(
 new TransformedMap<>(
 IterableWithWindowedValuesToIterable.of(), 
resultMap;
-multimap = HashMultimap.create();
+multimap = ArrayListMultimap.create();
   }
 
   multimap.put(
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
index b401d6d4e6d..2418825da20 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
@@ -568,6 +568,12 @@ public void testToMultimapDoFn() throws Exception {
 1,
 (Iterable>>>)
 ImmutableList.of(
+KV.of(
+windowA,
+WindowedValue.of(
+KV.of(1L, 11L), new Instant(3), windowA, 
PaneInfo.NO_FIRING)),
+// [BEAM-5184] Specifically test with a duplicate 
value to ensure that
+// duplicate key/values are not lost.
 KV.of(
 windowA,
 WindowedValue.of(
@@ -613,7 +619,7 @@ public void testToMultimapDoFn() throws Exception {
 
 outputMap = output.get(0).getValue().getValue();
 assertEquals(2, outputMap.size());
-assertThat(outputMap.get(1L), containsInAnyOrder(11L, 12L));
+assertThat(outputMap.get(1L), containsInAnyOrder(11L, 11L, 12L));
 assertThat(outputMap.get(2L), containsInAnyOrder(21L));
 
 outputMap = output.get(1).getValue().getValue();
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
index b275e03c57f..906ddecf72b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
@@ -18,7 +18,7 @@
 package org.apache.beam.sdk.values;
 
 import com.google.common.base.MoreObjects;
-import com.google.common.collect.HashMultimap;
+import com.google.common.collect.ArrayListMultimap;
 impo

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137865
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:57
Start Date: 24/Aug/18 15:57
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212675392
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 ##
 @@ -148,30 +147,45 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
 Review comment:
   Not needed since side input isn't supposed to be accessed before the window 
is ready. Added extra check in the handler and comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137865)
Time Spent: 6.5h  (was: 6h 20m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137863
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:57
Start Date: 24/Aug/18 15:57
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212675392
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 ##
 @@ -148,30 +147,45 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
 Review comment:
   Not needed since side input isn't supposed to be accessed before the window 
is ready. Added extra check.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137863)
Time Spent: 6h 20m  (was: 6h 10m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137867&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137867
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:58
Start Date: 24/Aug/18 15:58
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212675593
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java
 ##
 @@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.SideInputReference;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import 
org.apache.beam.runners.fnexecution.state.StateRequestHandlers.SideInputHandler;
+import 
org.apache.beam.runners.fnexecution.state.StateRequestHandlers.SideInputHandlerFactory;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.transforms.Materializations;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
+
+/**
+ * {@link StateRequestHandler} that uses {@link 
org.apache.beam.runners.core.SideInputHandler} to
+ * access the Flink broadcast state that represents side inputs.
+ */
+public class FlinkStreamingSideInputHandlerFactory implements 
SideInputHandlerFactory {
+
+  // Map from side input id to global PCollection id.
+  private final Map> sideInputToCollection;
+  private final org.apache.beam.runners.core.SideInputHandler runnerHandler;
+
+  /**
+   * Creates a new state handler for the given stage. Note that this requires 
a traversal of the
+   * stage itself, so this should only be called once per stage rather than 
once per bundle.
+   */
+  public static FlinkStreamingSideInputHandlerFactory forStage(
+  ExecutableStage stage,
+  Map> viewMapping,
+  org.apache.beam.runners.core.SideInputHandler runnerHandler) {
+ImmutableMap.Builder> sideInputBuilder = 
ImmutableMap.builder();
+for (SideInputReference sideInput : stage.getSideInputs()) {
+  SideInputId sideInputId =
+  SideInputId.newBuilder()
+  .setTransformId(sideInput.transform().getId())
+  .setLocalName(sideInput.localName())
+  .build();
+  sideInputBuilder.put(
+  sideInputId,
+  checkNotNull(
+  viewMapping.get(sideInputId),
+  "No side input for %s/%s",
+  sideInputId.getTransformId(),
+  sideInputId.getLocalName()));
+}
+
+FlinkStreamingSideInputHandlerFactory factory =
+new FlinkStreamingSideInputHandlerFactory(sideInputBuilder.build(), 
runnerHandler);
+return factory;
+  }
+
+  private FlinkStreamingSideInputHandlerFactory(
+  Map> sideInputToCollection,
+  org.apache.beam.runners.core.SideInputHandler runnerHandler) {
+this.sideInputToCollection = sideInputToCollection;
+this.runnerHandler = runnerHandler;
+  }
+
+  @Override
+  public  SideInputHandler forSideInput(
+  String transformId,
+  String sideInputId,
+  RunnerApi.FunctionSpec accessPattern,
+  Coder elementCoder

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137868&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137868
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:58
Start Date: 24/Aug/18 15:58
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6208: 
[BEAM-2930] Side input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#discussion_r212675673
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
 ##
 @@ -149,6 +149,7 @@ public static FlinkBatchPortablePipelineTranslator 
createTranslator() {
 translatorMap.put(
 PTransformTranslation.RESHUFFLE_URN,
 FlinkBatchPortablePipelineTranslator::translateReshuffle);
+// TODO: this does not seem required
 translatorMap.put(
 PTransformTranslation.CREATE_VIEW_TRANSFORM_URN,
 FlinkBatchPortablePipelineTranslator::translateView);
 
 Review comment:
   Dead code, removed it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137868)
Time Spent: 7h  (was: 6h 50m)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 7h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137870&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137870
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:59
Start Date: 24/Aug/18 15:59
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212675908
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount 
--job_endpoint=localhost:8099
 
 Review comment:
   Ah yes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137870)
Time Spent: 5h 40m  (was: 5.5h)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5184.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=137871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137871
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 24/Aug/18 15:59
Start Date: 24/Aug/18 15:59
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6257: [BEAM-5184] Multimap 
side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257#issuecomment-415803444
 
 
   Thanks for your contribution. This should make it out into the 2.7.0 release 
which is starting soon.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137871)
Time Spent: 5h 20m  (was: 5h 10m)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137872
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:01
Start Date: 24/Aug/18 16:01
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212676518
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount 
--job_endpoint=localhost:8099
 
 Review comment:
   `Unknown command-line option '--job_endpoint`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137872)
Time Spent: 5h 50m  (was: 5h 40m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137874
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:05
Start Date: 24/Aug/18 16:05
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212677632
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount 
--job_endpoint=localhost:8099
 
 Review comment:
   like this
   ``` 
   if (project.hasProperty("flinkMasterUrl"))
   args += ["--flink-master-url=${project.property('flinkMasterUrl')}"]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137874)
Time Spent: 6h  (was: 5h 50m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137876&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137876
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:14
Start Date: 24/Aug/18 16:14
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212679944
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount 
--job_endpoint=localhost:8099
 
 Review comment:
   Updated to include a property.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137876)
Time Spent: 6h 10m  (was: 6h)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137880
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:21
Start Date: 24/Aug/18 16:21
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212681815
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
 
 Review comment:
   This command doesn't work (not sure why actually), but this here does: 
`./gradlew -p sdks/python/container docker`
   
   (see python container build above)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137880)
Time Spent: 6h 20m  (was: 6h 10m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137882
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:30
Start Date: 24/Aug/18 16:30
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212684030
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount -PJobEndpoint=localhost
 
 Review comment:
   nit: the existing properties are usually lowerCamelCase
   
   What if I want to specify a port, is that no longer supported? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137882)
Time Spent: 6.5h  (was: 6h 20m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=137884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137884
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:33
Start Date: 24/Aug/18 16:33
Worklog Time Spent: 10m 
  Work Description: VaclavPlajt commented on issue #6257: [BEAM-5184] 
Multimap side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257#issuecomment-415812948
 
 
   Great. 👍 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137884)
Time Spent: 5.5h  (was: 5h 20m)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5171) org.apache.beam.sdk.io.CountingSourceTest.test[Un]boundedSourceSplits tests are flaky in Spark runner

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591876#comment-16591876
 ] 

Andrew Pilloud commented on BEAM-5171:
--

https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1333/testReport/junit/org.apache.beam.runners.spark.aggregators.metrics.sink/SparkMetricsSinkTest/testInBatchMode/

> org.apache.beam.sdk.io.CountingSourceTest.test[Un]boundedSourceSplits tests 
> are flaky in Spark runner
> -
>
> Key: BEAM-5171
> URL: https://issues.apache.org/jira/browse/BEAM-5171
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Valentyn Tymofieiev
>Assignee: Etienne Chauchot
>Priority: Major
>
> Two tests: 
>  org.apache.beam.sdk.io.CountingSourceTest.testUnboundedSourceSplits 
>  org.apache.beam.sdk.io.CountingSourceTest.testBoundedSourceSplits
> failed in a PostCommit [Spark Validates Runner test 
> suite|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/1277/testReport/]
>  with an error that seems to be common for Spark. Could this be due to 
> misconfiguration of Spark cluster? 
> Task serialization failed: java.io.IOException: Failed to create local dir in 
> /tmp/blockmgr-de91f449-e5d1-4be4-acaa-3ee06fdfa95b/1d.
>  java.io.IOException: Failed to create local dir in 
> /tmp/blockmgr-de91f449-e5d1-4be4-acaa-3ee06fdfa95b/1d.
>  at 
> org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:70)
>  at org.apache.spark.storage.DiskStore.remove(DiskStore.scala:116)
>  at 
> org.apache.spark.storage.BlockManager.removeBlockInternal(BlockManager.scala:1511)
>  at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1045)
>  at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083)
>  at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:841)
>  at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1404)
>  at 
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:123)
>  at 
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:88)
>  at 
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
>  at 
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
>  at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1482)
>  at 
> org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1039)
>  at 
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:947)
>  at 
> org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:891)
>  at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1780)
>  at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1772)
>  at 
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1761)
>  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137887
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:35
Start Date: 24/Aug/18 16:35
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212685454
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
 
 Review comment:
   Ah, there is a Unicode character in beam `®`. No idea how it ended up there.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137887)
Time Spent: 6h 50m  (was: 6h 40m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137885
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:35
Start Date: 24/Aug/18 16:35
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212685321
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount -PJobEndpoint=localhost
 
 Review comment:
   Will change to lowerCamelCase.
   
   You can still specify the port via `--job_port` but it is not exposed as a 
property yet. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137885)
Time Spent: 6h 40m  (was: 6.5h)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137889
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:37
Start Date: 24/Aug/18 16:37
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212685952
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount -PJobEndpoint=localhost
 
 Review comment:
   Why not support --job-endpoint in the previous form, that seems more 
intuitive?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137889)
Time Spent: 7h  (was: 6h 50m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1341

[Apache Jenkins Server]

See 


Changes:

[vaclav.plajt] [BEAM-5184] Side inputs aggregating multimap changed from 
`HashMultimap`

[lcwik] Fix-up BatchViewOverrides.java to use list based multimap.

--
[...truncated 18.18 MB...]
Submitted job: 2018-08-24_09_37_05-5542443573316172583

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Aug 24, 2018 4:37:06 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-08-24_09_37_05-5542443573316172583
Aug 24, 2018 4:37:06 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-08-24_09_37_05-5542443573316172583 with 0 
expected assertions.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:05.413Z: Autoscaling is enabled for job 
2018-08-24_09_37_05-5542443573316172583. The number of workers will be between 
1 and 1000.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:05.452Z: Autoscaling was automatically enabled for 
job 2018-08-24_09_37_05-5542443573316172583.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:08.399Z: Checking required Cloud APIs are enabled.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:08.711Z: Checking permissions granted to controller 
Service Account.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:13.437Z: Worker configuration: n1-standard-1 in 
us-central1-f.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:13.923Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.175Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.222Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.491Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.534Z: Elided trivial flatten 
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.571Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.612Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.641Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.683Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 24, 2018 4:37:17 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T16:37:14.729Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.

[jira] [Commented] (BEAM-5207) :beam-runners-apex:compileTestJava dependency issue

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591886#comment-16591886
 ] 

Andrew Pilloud commented on BEAM-5207:
--

This is probably the same as https://issues.apache.org/jira/browse/BEAM-5035

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1332/]

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1334/]

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1339/]

 

> :beam-runners-apex:compileTestJava dependency issue
> ---
>
> Key: BEAM-5207
> URL: https://issues.apache.org/jira/browse/BEAM-5207
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>
> Build failed without much in the way of errors, it appears to be some sort of 
> race condition. 
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1321
>  
> All input files are considered out-of-date for incremental task 
> ':beam-runners-apex:compileTestJava'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591887#comment-16591887
 ] 

Andrew Pilloud commented on BEAM-5035:
--

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1331/]

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1335/]

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1337/]

[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1338/]

 

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591898#comment-16591898
 ] 

Andrew Pilloud commented on BEAM-5035:
--

https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1341/

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5207) :beam-runners-apex:compileTestJava dependency issue

[Andrew Pilloud (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591897#comment-16591897
 ] 

Andrew Pilloud commented on BEAM-5207:
--

[https://builds.apache.org/job/beam_PreCommit_Java_Cron/263/]

https://builds.apache.org/job/beam_PreCommit_Java_Cron/264/

> :beam-runners-apex:compileTestJava dependency issue
> ---
>
> Key: BEAM-5207
> URL: https://issues.apache.org/jira/browse/BEAM-5207
> Project: Beam
>  Issue Type: New Feature
>  Components: build-system
>Reporter: Andrew Pilloud
>Assignee: Luke Cwik
>Priority: Major
>
> Build failed without much in the way of errors, it appears to be some sort of 
> race condition. 
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1321
>  
> All input files are considered out-of-date for incremental task 
> ':beam-runners-apex:compileTestJava'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5092) Nexmark 10x performance regression

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5092?focusedWorklogId=137891&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137891
 ]

ASF GitHub Bot logged work on BEAM-5092:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:50
Start Date: 24/Aug/18 16:50
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6268: [BEAM-5092] Prevent 
hash-lookup of schema on every record
URL: https://github.com/apache/beam/pull/6268#issuecomment-415817179
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137891)
Time Spent: 8h 40m  (was: 8.5h)

> Nexmark 10x performance regression
> --
>
> Key: BEAM-5092
> URL: https://issues.apache.org/jira/browse/BEAM-5092
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Andrew Pilloud
>Assignee: Reuven Lax
>Priority: Critical
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> There looks to be a 10x performance hit on the DirectRunner and Flink nexmark 
> jobs. It first showed up in this build:
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Direct/151/changes]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424]
> [https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137893
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 16:54
Start Date: 24/Aug/18 16:54
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212690050
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :b®eam-runners-flink_2.11-job-server-container docker
+//
+//  b) Otherwise, start a local JobService, for example, the Portable Flink 
runner
 //(in a separate shell since it continues to run):
 //
 //./gradlew :beam-runners-flink_2.11-job-server:runShadow
 //
 // Then you can run this example:
 //
+//  Docker (2a): 
+//
 //./gradlew :beam-sdks-python:portableWordCount
 //
+//  Local JobService (2b):
+//
+//./gradlew :beam-sdks-python:portableWordCount -PJobEndpoint=localhost
 
 Review comment:
   (this is more related to PortableOptions on the Python side, not what 
FlinkJobServerDriver does)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137893)
Time Spent: 7h 10m  (was: 7h)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137896&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137896
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:07
Start Date: 24/Aug/18 17:07
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212693575
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :beam-runners-flink_2.11-job-server-container docker
 
 Review comment:
   Still does not work: `Task 'beam-runners-flink_2.11-job-server-container' 
not found in root project 'beam'.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137896)
Time Spent: 7h 20m  (was: 7h 10m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137897
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:08
Start Date: 24/Aug/18 17:08
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212693774
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :beam-runners-flink_2.11-job-server-container docker
 
 Review comment:
   while this does: `./gradlew -p runners/flink/job-server-container docker`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137897)
Time Spent: 7.5h  (was: 7h 20m)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137898&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137898
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:10
Start Date: 24/Aug/18 17:10
Worklog Time Spent: 10m 
  Work Description: mxm commented on a change in pull request #6265: 
[BEAM-4130] Bring up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265#discussion_r212694411
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -204,19 +204,30 @@ task directRunnerIT(dependsOn: 'installGcpTest') {
 
 // Before running this, you need to:
 //
-// 1. build the SDK container:
+// 1. Build the SDK container:
 //
 //./gradlew -p sdks/python/container docker
 //
-// 2. start a local JobService, for example, the Portable Flink runner
+// 2. Either a) or b)
+//  a) If you want the Job Server to run in a Docker container:
+//
+//./gradlew :beam-runners-flink_2.11-job-server-container docker
 
 Review comment:
   Should be `./gradlew :beam-runners-flink_2.11-job-server-container:docker`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137898)
Time Spent: 7h 40m  (was: 7.5h)

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137904
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:32
Start Date: 24/Aug/18 17:32
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415828302
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137904)
Time Spent: 20m  (was: 10m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137903
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:32
Start Date: 24/Aug/18 17:32
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274
 
 
   Parallel builds have been very flaky for the last few weeks. For example, 
the post commit has failed in 12 of the last 16 builds due to this issue. We 
should disable parallelism until it can be fixed.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137903)
Time Spent: 10m
Remaining Estimate: 0h

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failure

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137905&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137905
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:36
Start Date: 24/Aug/18 17:36
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415829415
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137905)
Time Spent: 0.5h  (was: 20m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5180) Broken FileResultCoder via parseSchema change

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591973#comment-16591973
 ] 

Ankur Goenka commented on BEAM-5180:


>From the java.net.URI docs,

A hierarchical URI is subject to further parsing according to the syntax
{quote}[_scheme_{{*:*}}][{{*//*}}_authority_][_path_][{{*?*}}_query_][{{*#*}}_fragment_]{quote}
Which enforces  //

But to support HDFS and unblock our selves we should go with the rollback.

> Broken FileResultCoder via parseSchema change
> -
>
> Key: BEAM-5180
> URL: https://issues.apache.org/jira/browse/BEAM-5180
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Jozef Vilcek
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Recently this commit
> [https://github.com/apache/beam/commit/3fff58c21f94415f3397e185377e36d3df662384]
> introduced more strict schema parsing which is breaking the contract between 
> _FileResultCoder_ and _FileSystems.matchNewResource()_.
> Coder takes _ResourceId_ and serialize it via `_toString_` methods and then 
> relies on filesystem being able to parse it back again. Having strict 
> _scheme://_ breaks this at least for Hadoop filesystem which use _URI_ for 
> _ResourceId_ and produce _toString()_ in form of `_hdfs:/some/path_`
> I guess the _ResourceIdCoder_ is suffering the same problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5180) Broken FileResultCoder via parseSchema change

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5180?focusedWorklogId=137909&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137909
 ]

ASF GitHub Bot logged work on BEAM-5180:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:40
Start Date: 24/Aug/18 17:40
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6251: [BEAM-5180] Relax 
back restriction on parsing file scheme
URL: https://github.com/apache/beam/pull/6251#issuecomment-415830568
 
 
   From the java.net.URI docs,
   
   A hierarchical URI is subject to further parsing according to the syntax
   
   `[scheme:][//authority][path][?query][#fragment]`
   Which enforces  //
   
   But to support HDFS and unblock our selves we should go with the rollback.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137909)
Time Spent: 2h 10m  (was: 2h)

> Broken FileResultCoder via parseSchema change
> -
>
> Key: BEAM-5180
> URL: https://issues.apache.org/jira/browse/BEAM-5180
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Jozef Vilcek
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Recently this commit
> [https://github.com/apache/beam/commit/3fff58c21f94415f3397e185377e36d3df662384]
> introduced more strict schema parsing which is breaking the contract between 
> _FileResultCoder_ and _FileSystems.matchNewResource()_.
> Coder takes _ResourceId_ and serialize it via `_toString_` methods and then 
> relies on filesystem being able to parse it back again. Having strict 
> _scheme://_ breaks this at least for Hadoop filesystem which use _URI_ for 
> _ResourceId_ and produce _toString()_ in form of `_hdfs:/some/path_`
> I guess the _ResourceIdCoder_ is suffering the same problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5189) Field access of nested Row

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-5189:
---
Description: 
Right now, the field access to nested row does not work for any access to sub 
row's field.

 

We need to support SELECT row.row...row.field.

  was:
Right now, the field access to nested row does not work for any access to sub 
row's field.

 

We need to support it.


> Field access of nested Row
> --
>
> Key: BEAM-5189
> URL: https://issues.apache.org/jira/browse/BEAM-5189
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>
> Right now, the field access to nested row does not work for any access to sub 
> row's field.
>  
> We need to support SELECT row.row...row.field.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5121) Investigate flattening issue of nested Row

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5121?focusedWorklogId=137911&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137911
 ]

ASF GitHub Bot logged work on BEAM-5121:


Author: ASF GitHub Bot
Created on: 24/Aug/18 17:48
Start Date: 24/Aug/18 17:48
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6246: [BEAM-5121] Test 
complex types of Row and Array
URL: https://github.com/apache/beam/pull/6246#issuecomment-415832623
 
 
   Added the failed nested row field access case with `@Ignore` and JIRA


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137911)
Time Spent: 1.5h  (was: 1h 20m)

> Investigate flattening issue of nested Row
> --
>
> Key: BEAM-5121
> URL: https://issues.apache.org/jira/browse/BEAM-5121
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (2b777a1 -> f9d3f55)

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2b777a1  [BEAM-5184] Multimap side inputs with duplicate keys and 
values are being lost
 add 1a55497  Relax back restriction on parsing file scheme
 new f9d3f55  Merge pull request #6251 from JozoVilcek/relax-scheme-parsing

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java| 2 +-
 .../java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5180) Broken FileResultCoder via parseSchema change

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5180?focusedWorklogId=137916&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137916
 ]

ASF GitHub Bot logged work on BEAM-5180:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:07
Start Date: 24/Aug/18 18:07
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6251: [BEAM-5180] Relax 
back restriction on parsing file scheme
URL: https://github.com/apache/beam/pull/6251#issuecomment-415838093
 
 
   This is effectively a partial rollback of 
https://github.com/apache/beam/pull/5808
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137916)
Time Spent: 2h 20m  (was: 2h 10m)

> Broken FileResultCoder via parseSchema change
> -
>
> Key: BEAM-5180
> URL: https://issues.apache.org/jira/browse/BEAM-5180
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Jozef Vilcek
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Recently this commit
> [https://github.com/apache/beam/commit/3fff58c21f94415f3397e185377e36d3df662384]
> introduced more strict schema parsing which is breaking the contract between 
> _FileResultCoder_ and _FileSystems.matchNewResource()_.
> Coder takes _ResourceId_ and serialize it via `_toString_` methods and then 
> relies on filesystem being able to parse it back again. Having strict 
> _scheme://_ breaks this at least for Hadoop filesystem which use _URI_ for 
> _ResourceId_ and produce _toString()_ in form of `_hdfs:/some/path_`
> I guess the _ResourceIdCoder_ is suffering the same problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6251 from JozoVilcek/relax-scheme-parsing

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f9d3f558ee78a0d3a43bec664235579a2a5f98b4
Merge: 2b777a1 1a55497
Author: Andrew Pilloud 
AuthorDate: Fri Aug 24 11:07:59 2018 -0700

Merge pull request #6251 from JozoVilcek/relax-scheme-parsing

[BEAM-5180] Relax back restriction on parsing file scheme

 sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java| 2 +-
 .../java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5180) Broken FileResultCoder via parseSchema change

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5180?focusedWorklogId=137917&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137917
 ]

ASF GitHub Bot logged work on BEAM-5180:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:08
Start Date: 24/Aug/18 18:08
Worklog Time Spent: 10m 
  Work Description: apilloud closed pull request #6251: [BEAM-5180] Relax 
back restriction on parsing file scheme
URL: https://github.com/apache/beam/pull/6251
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
index be89c9ec099..7de41c1174a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
@@ -69,7 +69,7 @@
 
   public static final String DEFAULT_SCHEME = "file";
   private static final Pattern FILE_SCHEME_PATTERN =
-  Pattern.compile("(?[a-zA-Z][-a-zA-Z0-9+.]*)://.*");
+  Pattern.compile("(?[a-zA-Z][-a-zA-Z0-9+.]*):/.*");
   private static final Pattern GLOB_PATTERN = Pattern.compile("[*?{}]");
 
   private static final AtomicReference> 
SCHEME_TO_FILESYSTEM =
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
index 22f71f6e09f..0fbeb71325d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
@@ -196,6 +196,7 @@ public void testValidMatchNewResourceForLocalFileSystem() {
   @Test(expected = IllegalArgumentException.class)
   public void testInvalidSchemaMatchNewResource() {
 assertEquals("file", 
FileSystems.matchNewResource("invalidschema://tmp/f1", false));
+assertEquals("file", FileSystems.matchNewResource("c:/tmp/f1", false));
   }
 
   private List toResourceIds(List paths, final boolean 
isDirectory) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137917)
Time Spent: 2.5h  (was: 2h 20m)

> Broken FileResultCoder via parseSchema change
> -
>
> Key: BEAM-5180
> URL: https://issues.apache.org/jira/browse/BEAM-5180
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Jozef Vilcek
>Assignee: Kenneth Knowles
>Priority: Blocker
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Recently this commit
> [https://github.com/apache/beam/commit/3fff58c21f94415f3397e185377e36d3df662384]
> introduced more strict schema parsing which is breaking the contract between 
> _FileResultCoder_ and _FileSystems.matchNewResource()_.
> Coder takes _ResourceId_ and serialize it via `_toString_` methods and then 
> relies on filesystem being able to parse it back again. Having strict 
> _scheme://_ breaks this at least for Hadoop filesystem which use _URI_ for 
> _ResourceId_ and produce _toString()_ in form of `_hdfs:/some/path_`
> I guess the _ResourceIdCoder_ is suffering the same problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137919
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:19
Start Date: 24/Aug/18 18:19
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415841294
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137919)
Time Spent: 40m  (was: 0.5h)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #266

[Apache Jenkins Server]

See 


Changes:

[vaclav.plajt] [BEAM-5184] Side inputs aggregating multimap changed from 
`HashMultimap`

[lcwik] Fix-up BatchViewOverrides.java to use list based multimap.

[rpathak] [BEAM-3026] Adding retrying behavior on ESIO for error http 429 (Too

--
[...truncated 17.10 MB...]
INFO: 2018-08-24T18:18:58.272Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.304Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.340Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.385Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.431Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.474Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.518Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.564Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.611Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.653Z: Unzipping flatten s13-u58 for input 
s14.org.apache.beam.sdk.values.PCollection.:384#43d0369d0f0559b7-c56
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.689Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign,
 through flatten s13-u58, into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.727Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
Aug 24, 2018 6:19:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-24T18:18:58.775Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write

[beam] 01/01: Merge pull request #6265: [BEAM-4130] Bring up Job Server container for Python jobs

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 96d2887f3d807d3546339eb83749feadc02dcd9c
Merge: f9d3f55 5b3b9e0
Author: Thomas Weise 
AuthorDate: Fri Aug 24 11:23:29 2018 -0700

Merge pull request #6265: [BEAM-4130] Bring up Job Server container for 
Python jobs

 runners/flink/job-server-container/Dockerfile  |   2 +-
 runners/flink/job-server-container/build.gradle|   2 +-
 runners/flink/job-server/build.gradle  |   2 +-
 .../beam/runners/flink/FlinkJobServerDriver.java   |  50 +++---
 .../beam/runners/fnexecution/ServerFactory.java|  21 -
 .../control/DockerJobBundleFactory.java|  22 -
 .../environment/DockerEnvironmentFactory.java  |   2 +
 .../runners/portability/flink_runner_test.py   |   3 +-
 .../apache_beam/runners/portability/job_server.py  | 101 +
 .../runners/portability/portable_runner.py |  33 +--
 sdks/python/build.gradle   |  18 +++-
 11 files changed, 224 insertions(+), 32 deletions(-)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137921&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137921
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:23
Start Date: 24/Aug/18 18:23
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415842451
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137921)
Time Spent: 50m  (was: 40m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137920&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137920
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:23
Start Date: 24/Aug/18 18:23
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6265: [BEAM-4130] Bring up 
Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/flink/job-server-container/Dockerfile 
b/runners/flink/job-server-container/Dockerfile
index a9aff21b6d6..569c2ab0406 100644
--- a/runners/flink/job-server-container/Dockerfile
+++ b/runners/flink/job-server-container/Dockerfile
@@ -19,7 +19,7 @@
 FROM openjdk:8
 MAINTAINER "Apache Beam "
 
-ADD target/beam-runners-flink_2.11-job-server.jar /opt/apache/beam/jars/
+ADD target/beam-runners-flink-job-server.jar /opt/apache/beam/jars/
 ADD target/flink-job-server.sh /opt/apache/beam/
 
 WORKDIR /opt/apache/beam
diff --git a/runners/flink/job-server-container/build.gradle 
b/runners/flink/job-server-container/build.gradle
index 4d5f53316e3..d2b026804d7 100644
--- a/runners/flink/job-server-container/build.gradle
+++ b/runners/flink/job-server-container/build.gradle
@@ -38,7 +38,7 @@ dependencies {
 task copyDockerfileDependencies(type: Copy) {
   // Required Jars
   from configurations.dockerDependency
-  rename 'beam-runners-flink_2.11-job-server.*.jar', 
'beam-runners-flink_2.11-job-server.jar'
+  rename 'beam-runners-flink_2.11-job-server.*.jar', 
'beam-runners-flink-job-server.jar'
   into "build/target"
   // Entry script
   from file("./flink-job-server.sh")
diff --git a/runners/flink/job-server/build.gradle 
b/runners/flink/job-server/build.gradle
index 5e1e01ae74d..ee6b70f1f37 100644
--- a/runners/flink/job-server/build.gradle
+++ b/runners/flink/job-server/build.gradle
@@ -55,7 +55,7 @@ dependencies {
 // task will not work because the flink runner classes only exist in the shadow
 // jar.
 runShadow {
-  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost:8099"
+  def jobHost = project.hasProperty("jobHost") ? project.property("jobHost") : 
"localhost"
   def artifactsDir = project.hasProperty("artifactsDir") ?  
project.property("artifactsDir") : "/tmp/flink-artifacts"
   def cleanArtifactsPerJob = project.hasProperty("cleanArtifactsPerJob")
   args = ["--job-host=${jobHost}", "--artifacts-dir=${artifactsDir}"]
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
index 7e9b14a2acd..13f48c53018 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
@@ -45,15 +45,22 @@
 
   private final ListeningExecutorService executor;
   private final ServerConfiguration configuration;
-  private final ServerFactory serverFactory;
+  private final ServerFactory jobServerFactory;
+  private final ServerFactory artifactServerFactory;
   private GrpcFnServer jobServer;
   private GrpcFnServer 
artifactStagingServer;
 
   /** Configuration for the jobServer. */
   public static class ServerConfiguration {
-@Option(name = "--job-host", usage = "The job server host string")
+@Option(name = "--job-host", usage = "The job server host name")
 private String host = "";
 
+@Option(name = "--job-port", usage = "The job service port. (Default: 
8099)")
+private int port = 8099;
+
+@Option(name = "--artifact-port", usage = "The artifact service port. 
(Default: 8098)")
+private int artifactPort = 8098;
+
 @Option(name = "--artifacts-dir", usage = "The location to store staged 
artifact files")
 private String artifactStagingPath = "/tmp/beam-artifact-staging";
 
@@ -100,24 +107,30 @@ public static FlinkJobServerDriver 
fromConfig(ServerConfiguration configuration)
 new 
ThreadFactoryBuilder().setNameFormat("flink-runner-job-server").setDaemon(true).build();
 ListeningExecutorService executor =
 
MoreExecutors.listeningDecorator(Executors.newCachedThreadPool(threadFactory));
-ServerFactory serverFactory = ServerFactory.createDefault();
-return create(configuration, executor, serverFactory);
+ServerFactory jobServerFactory = ServerFactory.createWithPortSupplier(() 
-> configuration.port);
+ServerFactory artifactServerFactory =
+ServerFactory.createWithPortSupplier(() -> configuration.artifactPort);
+retu

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137922
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:24
Start Date: 24/Aug/18 18:24
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6208: [BEAM-2930] Side input 
support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#issuecomment-415842617
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137922)
Time Spent: 7h 10m  (was: 7h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (f9d3f55 -> 96d2887)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f9d3f55  Merge pull request #6251 from JozoVilcek/relax-scheme-parsing
 add 5b3b9e0  [BEAM-4130] Bring up Job Server container for Python jobs
 new 96d2887  Merge pull request #6265: [BEAM-4130] Bring up Job Server 
container for Python jobs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/flink/job-server-container/Dockerfile  |   2 +-
 runners/flink/job-server-container/build.gradle|   2 +-
 runners/flink/job-server/build.gradle  |   2 +-
 .../beam/runners/flink/FlinkJobServerDriver.java   |  50 +++---
 .../beam/runners/fnexecution/ServerFactory.java|  21 -
 .../control/DockerJobBundleFactory.java|  22 -
 .../environment/DockerEnvironmentFactory.java  |   2 +
 .../runners/portability/flink_runner_test.py   |   3 +-
 .../apache_beam/runners/portability/job_server.py  | 101 +
 .../runners/portability/portable_runner.py |  33 +--
 sdks/python/build.gradle   |  18 +++-
 11 files changed, 224 insertions(+), 32 deletions(-)
 create mode 100644 sdks/python/apache_beam/runners/portability/job_server.py

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #784

[Apache Jenkins Server]

PKgQIGUAB"
}
  ]
}
  ],
  "is_stream_like": true
}
  ],
  "is_pair_like": true
},
{
  "@type": "kind:global_window"
}
  ],
  "is_wrapper": true
}
  }
],
"parallel_input": {
  "@type": "OutputReference",
  "step_name": "e7",
  "output_name": "i0"
},
"serialized_fn": 
"%0A%29%22%27%0A%02c1%12%21%0A%1F%0A%1D%0A%1Bbeam:coder:global_window:v1j9%0A%25%0A%23%0A%21beam:windowfn:global_windows:v0.1%10%01%1A%02c1%22%02:%00%28%010%018%02H%01"
  }
},
{
  "kind": "ParallelDo",
  "name": "e9",
  "properties": {
"user_name": "passert.Sum(emit2)/passert.sumFn'2",
"output_info": [
  {
"user_name": "bogus",
"output_name": "bogus",
"encoding": {
  "@type": "kind:windowed_value",
  "component_encodings": [
{
  "@type": "kind:bytes"
},
{
  "@type": "kind:global_window"
}
  ],
  "is_wrapper": true
}
  }
],
"parallel_input": {
  "@type": "OutputReference",
  "step_name": "e8",
  "output_name": "i0"
},
"serialized_fn": "e9"
  }
}
  ],
  "type": "JOB_TYPE_BATCH"
}
2018/08/24 18:27:41 Submitted job: 2018-08-24_11_27_40-18225356857618668520
2018/08/24 18:27:41 Console: 
https://console.cloud.google.com/dataflow/job/2018-08-24_11_27_40-18225356857618668520?project=apache-beam-testing
2018/08/24 18:27:41 Logs: 
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-08-24_11_27_40-18225356857618668520
2018/08/24 18:27:41 Submitted job: 2018-08-24_11_27_40-6446445893504212913
2018/08/24 18:27:41 Console: 
https://console.cloud.google.com/dataflow/job/2018-08-24_11_27_40-6446445893504212913?project=apache-beam-testing
2018/08/24 18:27:41 Logs: 
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-08-24_11_27_40-6446445893504212913
2018/08/24 18:27:41 Job state: JOB_STATE_PENDING ...
2018/08/24 18:27:41 Job state: JOB_STATE_PENDING ...
2018/08/24 18:27:41 Submitted job: 2018-08-24_11_27_40-9246062846768992572
2018/08/24 18:27:41 Console: 
https://console.cloud.google.com/dataflow/job/2018-08-24_11_27_40-9246062846768992572?project=apache-beam-testing
2018/08/24 18:27:41 Logs: 
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-08-24_11_27_40-9246062846768992572
2018/08/24 18:27:41 Job state: JOB_STATE_PENDING ...
2018/08/24 18:27:41 Submitted job: 2018-08-24_11_27_40-5718962993292897696
2018/08/24 18:27:41 Console: 
https://console.cloud.google.com/dataflow/job/2018-08-24_11_27_40-5718962993292897696?project=apache-beam-testing
2018/08/24 18:27:41 Logs: 
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-08-24_11_27_40-5718962993292897696
2018/08/24 18:27:41 Submitted job: 2018-08-24_11_27_40-17435788609163937168
2018/08/24 18:27:41 Console: 
https://console.cloud.google.com/dataflow/job/2018-08-24_11_27_40-17435788609163937168?project=apache-beam-testing
2018/08/24 18:27:41 Logs: 
https://console.cloud.google.com/logs/viewer?project=apache-beam-testing&resource=dataflow_step%2Fjob_id%2F2018-08-24_11_27_40-17435788609163937168
2018/08/24 18:27:41 Job state: JOB_STATE_PENDING ...
2018/08/24 18:27:41 Job state: JOB_STATE_PENDING ...
2018/08/24 18:28:11 Job still running ...
2018/08/24 18:28:11 Job still running ...
2018/08/24 18:28:11 Job still running ...
2018/08/24 18:28:11 Job still running ...
2018/08/24 18:28:11 Job still running ...
2018/08/24 18:28:41 Job still running ...
2018/08/24 18:28:41 Job still running ...
2018/08/24 18:28:41 Job still running ...
2018/08/24 18:28:41 Job still running ...
2018/08/24 18:28:41 Job still running ...
2018/08/24 18:29:11 Job still running ...
2018/08/24 18:29:11 Job still running ...
2018/08/24 18:29:11 Job still running ...
2018/08/24 18:29:11 Job still running ...
2018/08/24 18:29:11 Job still running ...
2018/08/24 18:29:4

[jira] [Assigned] (BEAM-5187) Create a ProcessJobBundleFactory for non-dockerized SDK harness

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise reassigned BEAM-5187:
--

Assignee: Maximilian Michels  (was: Kenneth Knowles)

> Create a ProcessJobBundleFactory for non-dockerized SDK harness
> ---
>
> Key: BEAM-5187
> URL: https://issues.apache.org/jira/browse/BEAM-5187
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> As discussed on the mailing list [1], we want to giver users an option to 
> execute portable pipelines without Docker. Analog to the 
> {{DockerJobBundleFactory}}, a {{ProcessJobBundleFactory}} could be added to 
> directly fork SDK harness processes.
> Artifacts will be provided by an artifact directory or could be setup similar 
> to the existing bootstrapping code ("boot.go") which we use for containers.
> The process-based execution can optionally be configured via the pipeline 
> options.
> [1] 
> [https://lists.apache.org/thread.html/d8b81e9f74f77d74c8b883cda80fa48efdcaf6ac2ad313c4fe68795a@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved BEAM-4130.

   Resolution: Fixed
Fix Version/s: 2.7.0

> Portable Flink runner JobService entry point in a Docker container
> --
>
> Key: BEAM-4130
> URL: https://issues.apache.org/jira/browse/BEAM-4130
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Maximilian Michels
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> The portable Flink runner exists as a Job Service that runs somewhere. We 
> need a main entry point that itself spins up the job service (and artifact 
> staging service). The main program itself should be packaged into an uberjar 
> such that it can be run locally or submitted to a Flink deployment via `flink 
> run`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5213) sdks/go/test/regression failed with no results

[Andrew Pilloud (JIRA)]

Andrew Pilloud created BEAM-5213:


 Summary: sdks/go/test/regression failed with no results
 Key: BEAM-5213
 URL: https://issues.apache.org/jira/browse/BEAM-5213
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Andrew Pilloud
Assignee: Henning Rohde


The details on which test failed are missing, but something failed. 
https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/784/

 
*11:26:20* Result of package 
github.com/apache/beam/sdks/go/test/regression:*11:26:20* Test for 
github.com/apache/beam/sdks/go/test/regression finished, 1 completed, 1 failed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6208: [BEAM-2930] Side input support for Flink portable streaming

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 88b3556ad99e16b3a63064d7243d1675df501ef5
Merge: 96d2887 178d258
Author: Thomas Weise 
AuthorDate: Fri Aug 24 11:58:16 2018 -0700

Merge pull request #6208: [BEAM-2930] Side input support for Flink portable 
streaming

 .../core/construction/RunnerPCollectionView.java   |  14 +-
 .../apache/beam/runners/core/SideInputHandler.java |  41 ++--
 .../FlinkBatchPortablePipelineTranslator.java  |  14 --
 .../FlinkStreamingPortablePipelineTranslator.java  | 241 +++--
 .../flink/FlinkStreamingTransformTranslators.java  |   2 +-
 .../beam/runners/flink/translation/.DS_Store   | Bin 0 -> 6148 bytes
 .../FlinkBatchExecutableStageContext.java  |   3 +-
 .../FlinkBatchSideInputHandlerFactory.java |   8 -
 .../functions/FlinkExecutableStageContext.java |   5 -
 .../functions/FlinkExecutableStageFunction.java|   3 +-
 .../FlinkStreamingSideInputHandlerFactory.java | 168 ++
 ...CountingFlinkExecutableStageContextFactory.java |   8 -
 .../wrappers/streaming/DoFnOperator.java   |  24 +-
 .../streaming/ExecutableStageDoFnOperator.java |  48 +++-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   5 +-
 .../FlinkExecutableStageFunctionTest.java  |   5 +-
 16 files changed, 494 insertions(+), 95 deletions(-)

[beam] branch master updated (96d2887 -> 88b3556)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 96d2887  Merge pull request #6265: [BEAM-4130] Bring up Job Server 
container for Python jobs
 add 178d258  [BEAM-2930] Side input support for Flink portable streaming.
 new 88b3556  Merge pull request #6208: [BEAM-2930] Side input support for 
Flink portable streaming

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/RunnerPCollectionView.java   |  14 +-
 .../apache/beam/runners/core/SideInputHandler.java |  41 ++--
 .../FlinkBatchPortablePipelineTranslator.java  |  14 --
 .../FlinkStreamingPortablePipelineTranslator.java  | 241 +++--
 .../flink/FlinkStreamingTransformTranslators.java  |   2 +-
 .../beam/runners/flink/translation/.DS_Store   | Bin 0 -> 6148 bytes
 .../FlinkBatchExecutableStageContext.java  |   3 +-
 .../FlinkBatchSideInputHandlerFactory.java |   8 -
 .../functions/FlinkExecutableStageContext.java |   5 -
 .../functions/FlinkExecutableStageFunction.java|   3 +-
 .../FlinkStreamingSideInputHandlerFactory.java | 168 ++
 ...CountingFlinkExecutableStageContextFactory.java |   8 -
 .../wrappers/streaming/DoFnOperator.java   |  24 +-
 .../streaming/ExecutableStageDoFnOperator.java |  48 +++-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   5 +-
 .../FlinkExecutableStageFunctionTest.java  |   5 +-
 16 files changed, 494 insertions(+), 95 deletions(-)
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/.DS_Store
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137936&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137936
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:58
Start Date: 24/Aug/18 18:58
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415851835
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137936)
Time Spent: 1h 10m  (was: 1h)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137934
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:58
Start Date: 24/Aug/18 18:58
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415851774
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137934)
Time Spent: 1h  (was: 50m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137935&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137935
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 24/Aug/18 18:58
Start Date: 24/Aug/18 18:58
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6208: [BEAM-2930] Side 
input support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java
index c488522aa64..c453329051d 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RunnerPCollectionView.java
@@ -34,22 +34,22 @@
 import org.apache.beam.sdk.values.WindowingStrategy;
 
 /** A {@link PCollectionView} created from the components of a {@link 
SideInput}. */
-class RunnerPCollectionView extends PValueBase implements 
PCollectionView {
+public class RunnerPCollectionView extends PValueBase implements 
PCollectionView {
   private final TupleTag>> tag;
   private final ViewFn>, T> viewFn;
   private final WindowMappingFn windowMappingFn;
   private final @Nullable WindowingStrategy windowingStrategy;
-  private final @Nullable Coder>> coder;
-  private final transient PCollection pCollection;
+  private final @Nullable Coder coder;
+  private final transient @Nullable PCollection pCollection;
 
   /** Create a new {@link RunnerPCollectionView} from the provided components. 
*/
-  RunnerPCollectionView(
-  PCollection pCollection,
+  public RunnerPCollectionView(
+  @Nullable PCollection pCollection,
   TupleTag>> tag,
   ViewFn>, T> viewFn,
   WindowMappingFn windowMappingFn,
   @Nullable WindowingStrategy windowingStrategy,
-  @Nullable Coder>> coder) {
+  @Nullable Coder coder) {
 this.pCollection = pCollection;
 this.tag = tag;
 this.viewFn = viewFn;
@@ -84,7 +84,7 @@
   }
 
   @Override
-  public Coder>> getCoderInternal() {
+  public Coder getCoderInternal() {
 return coder;
   }
 
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
index e8c2b66ba42..1a6826d94b1 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
@@ -38,7 +38,6 @@
 import org.apache.beam.sdk.transforms.ViewFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollectionView;
 
 /**
@@ -148,30 +147,46 @@ public void addSideInputValue(PCollectionView 
sideInput, WindowedValue T get(PCollectionView view, BoundedWindow window) {
-@SuppressWarnings("unchecked")
-Coder windowCoder =
-(Coder) 
view.getWindowingStrategyInternal().getWindowFn().windowCoder();
-
-StateTag>> stateTag = 
sideInputContentsTags.get(view);
-
-ValueState> state =
-stateInternals.state(StateNamespaces.window(windowCoder, window), 
stateTag);
 
-// TODO: Add support for choosing which representation is contained based 
upon the
-// side input materialization. We currently can assume that we always have 
a multimap
-// materialization as that is the only supported type within the Java SDK.
-@Nullable Iterable> elements = (Iterable>) state.read();
+@Nullable Iterable elements = getIterable(view, window);
 
 if (elements == null) {
   elements = Collections.emptyList();
 }
 
+// TODO: Add support for choosing which representation is contained based 
upon the
+// side input materialization. We currently can assume that we always have 
a multimap
+// materialization as that is the only supported type within the Java SDK.
 ViewFn viewFn = (ViewFn) 
view.getViewFn();
 Coder keyCoder = ((KvCoder) 
view.getCoderInternal()).getKeyCoder();
 return (T)
 viewFn.apply(InMemoryMultimapSideInputView.fromIterable(keyCoder, 
(Iterable) elements));
   }
 
+  /**
+   * Retrieve the value as written by {@link 
#addSideInputValue(PCollectionView, WindowedValue)},
+   * without applying the SDK specific {

[jira] [Created] (BEAM-5214) Update Java quickstart to use maven

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-5214:
-

 Summary: Update Java quickstart to use maven
 Key: BEAM-5214
 URL: https://issues.apache.org/jira/browse/BEAM-5214
 Project: Beam
  Issue Type: Bug
  Components: examples-java, website
Reporter: Robert Bradshaw
Assignee: Reuven Lax


The existing quickstart still uses mvn commands. 

https://beam.apache.org/get-started/quickstart-java/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Go_GradleBuild #785

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137937
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:06
Start Date: 24/Aug/18 19:06
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415853979
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137937)
Time Spent: 1h 20m  (was: 1h 10m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137941&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137941
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:24
Start Date: 24/Aug/18 19:24
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415858719
 
 
   Run Java Precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137941)
Time Spent: 1.5h  (was: 1h 20m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5151) Add EXTERNAL to CREATE TABLE statement

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5151?focusedWorklogId=137942&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137942
 ]

ASF GitHub Bot logged work on BEAM-5151:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:31
Start Date: 24/Aug/18 19:31
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6252: 
[BEAM-5151][SQL] create external table
URL: https://github.com/apache/beam/pull/6252#discussion_r212732715
 
 

 ##
 File path: sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
 ##
 @@ -191,6 +191,18 @@ SqlCreate SqlCreateExternalTable() :
 }
 }
 
+SqlCreate SqlCreateTableNotSupportedMessage(Span s, boolean replace) :
+{
+
+}
+{
+  
+
+  {
+return new SqlCreateTableNotSupportedMessage(s.end(this), replace);
 
 Review comment:
   is it possible to throw from here directly without creating a new class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137942)
Time Spent: 2h 40m  (was: 2.5h)

> Add EXTERNAL to CREATE TABLE statement
> --
>
> Key: BEAM-5151
> URL: https://issues.apache.org/jira/browse/BEAM-5151
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> BeamSQL allows [CREATE 
> TABLE|https://beam.apache.org/documentation/dsls/sql/create-table/] 
> statements to register virtual tables from external storage systems (e.g. 
> BigQuery). 
>  
> BeamSQL is not a storage system, so any table registered by "CREATE TABLE" 
> statement is essentially equivalent to be registered by "CREATE EXTERNAL 
> TABLE", which requires the user to provide a LOCATION and BeamSQL will 
> register the table outside of current execution environment based on LOCATION.
>  
> So I propose to add EXTERNAL keyword to "CREATE TABLE" in BeamSQL to help 
> users understand they are registering tables, and BeamSQL does not create non 
> existing tables by running CREATE TABLE (at least on some storage systems, if 
> not all). 
>  
> We can make the EXTERNAL keyword either required or optional.
>  
> If we make the EXTERNAL keyword required:
>  
> Pros:
> a. We can get rid of the registering table semantic on CREATE TABLE. 
> b, We keep the room that we could add CREATE TABLE back in the future if we 
> want CREATE TABLE to create, rather than not only register tables in BeamSQL. 
>  
> Cons:
> 1. CREATE TABLE syntax will not be supported so existing BeamSQL pipelines 
> which has CREATE TABLE require changes.
> 2. It's required to type tedious EXTERNAL keyword every time, especially in 
> SQL Shell.
>  
> If we make the EXTERNAL keyword optional, we will have reversed pros and cons 
> above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1342

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=137943&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137943
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:46
Start Date: 24/Aug/18 19:46
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #4548: [BEAM-3310] Metrics 
pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-415863810
 
 
   @aljoscha 
   
   Would like to respond to your email here to connect contexts:
   
   > the accumulators not being available is an artifact of how detached mode 
is implemented. Basically, we fire of the job and then forget everything about 
it. That's why there is no way of getting any data about the job.
   Regarding Metrics, with Flink you can set up a Metrics reporter 
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html)
 and Beam metrics will then be exported like Flink metrics to that reporter.
   
   >Best,
   Aljoscha
   
   I understand that we forget about the job once we fire it in detached mode 
and also that the Beam metrics are available through a Metrics reporter from 
Flink.
   
   Now my understanding was that these metrics from Beam need to be accumulated 
by Flink first in order to report from across task managers etc. -> whether 
through metrics reporter or Flink UI. This is what @StephanEwen assumed too. My 
question is simple -> if Flink is actually accumulating metrics, why is it not 
shown in Flink UI for Beam jobs under "Accumulators"? Is this a bug or this 
support is not there from Flink? If support is not present then how are the 
metrics reported/accumulated through Metrics reporter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137943)
Time Spent: 16.5h  (was: 16h 20m)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 16.5h
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=137944&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137944
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:49
Start Date: 24/Aug/18 19:49
Worklog Time Spent: 10m 
  Work Description: zorro786 edited a comment on issue #4548: [BEAM-3310] 
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-415863810
 
 
   @aljoscha 
   
   Would like to respond to your email here to connect contexts:
   
   > the accumulators not being available is an artifact of how detached mode 
is implemented. Basically, we fire of the job and then forget everything about 
it. That's why there is no way of getting any data about the job.
   Regarding Metrics, with Flink you can set up a Metrics reporter 
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html)
 and Beam metrics will then be exported like Flink metrics to that reporter.
   
   >Best,
   Aljoscha
   
   I understand that we forget about the job once we fire it in detached mode 
(no way to get metrics from `PipelineResult` and also that the Beam metrics are 
available through a Metrics reporter from Flink.
   
   Now my understanding was that these metrics from Beam need to be accumulated 
by Flink first in order to report from across task managers etc. -> whether 
through metrics reporter or Flink UI. This is what @StephanEwen assumed too. My 
question is simple -> if Flink is actually accumulating metrics, why is it not 
shown in Flink UI for Beam jobs under "Accumulators"? Is this a bug or this 
support is not there from Flink? If support is not present then how are the 
metrics reported/accumulated through Metrics reporter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137944)
Time Spent: 16h 40m  (was: 16.5h)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=137945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137945
 ]

ASF GitHub Bot logged work on BEAM-3310:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:49
Start Date: 24/Aug/18 19:49
Worklog Time Spent: 10m 
  Work Description: zorro786 edited a comment on issue #4548: [BEAM-3310] 
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-415863810
 
 
   @aljoscha 
   
   Would like to respond to your email here to connect contexts:
   
   > the accumulators not being available is an artifact of how detached mode 
is implemented. Basically, we fire of the job and then forget everything about 
it. That's why there is no way of getting any data about the job.
   Regarding Metrics, with Flink you can set up a Metrics reporter 
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html)
 and Beam metrics will then be exported like Flink metrics to that reporter.
   
   >Best,
   Aljoscha
   
   I understand that we forget about the job once we fire it in detached mode 
(no way to get metrics from `PipelineResult`) and also that the Beam metrics 
are available through a Metrics reporter from Flink.
   
   Now my understanding was that these metrics from Beam need to be accumulated 
by Flink first in order to report from across task managers etc. -> whether 
through metrics reporter or Flink UI. This is what @StephanEwen assumed too. My 
question is simple -> if Flink is actually accumulating metrics, why is it not 
shown in Flink UI for Beam jobs under "Accumulators"? Is this a bug or this 
support is not there from Flink? If support is not present then how are the 
metrics reported/accumulated through Metrics reporter?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137945)
Time Spent: 16h 50m  (was: 16h 40m)

> Push metrics to a backend in an runner agnostic way
> ---
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-extensions-metrics, sdk-java-core
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics 
> (either at the end of the pipeline or while it runs) because they don't have 
> all the same capabilities towards metrics (e.g. spark runner configures sinks 
>  like csv, graphite or in memory sinks using the spark engine conf). The 
> target is to push the metrics in the common runner code so that no matter the 
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML: 
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5151) Add EXTERNAL to CREATE TABLE statement

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5151?focusedWorklogId=137948&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137948
 ]

ASF GitHub Bot logged work on BEAM-5151:


Author: ASF GitHub Bot
Created on: 24/Aug/18 19:55
Start Date: 24/Aug/18 19:55
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6252: 
[BEAM-5151][SQL] create external table
URL: https://github.com/apache/beam/pull/6252#discussion_r212738728
 
 

 ##
 File path: sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
 ##
 @@ -191,6 +191,18 @@ SqlCreate SqlCreateExternalTable() :
 }
 }
 
+SqlCreate SqlCreateTableNotSupportedMessage(Span s, boolean replace) :
+{
+
+}
+{
+  
+
+  {
+return new SqlCreateTableNotSupportedMessage(s.end(this), replace);
 
 Review comment:
   Yes it is. I changed the code to throw the exception and not create a new 
class.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137948)
Time Spent: 2h 50m  (was: 2h 40m)

> Add EXTERNAL to CREATE TABLE statement
> --
>
> Key: BEAM-5151
> URL: https://issues.apache.org/jira/browse/BEAM-5151
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> BeamSQL allows [CREATE 
> TABLE|https://beam.apache.org/documentation/dsls/sql/create-table/] 
> statements to register virtual tables from external storage systems (e.g. 
> BigQuery). 
>  
> BeamSQL is not a storage system, so any table registered by "CREATE TABLE" 
> statement is essentially equivalent to be registered by "CREATE EXTERNAL 
> TABLE", which requires the user to provide a LOCATION and BeamSQL will 
> register the table outside of current execution environment based on LOCATION.
>  
> So I propose to add EXTERNAL keyword to "CREATE TABLE" in BeamSQL to help 
> users understand they are registering tables, and BeamSQL does not create non 
> existing tables by running CREATE TABLE (at least on some storage systems, if 
> not all). 
>  
> We can make the EXTERNAL keyword either required or optional.
>  
> If we make the EXTERNAL keyword required:
>  
> Pros:
> a. We can get rid of the registering table semantic on CREATE TABLE. 
> b, We keep the room that we could add CREATE TABLE back in the future if we 
> want CREATE TABLE to create, rather than not only register tables in BeamSQL. 
>  
> Cons:
> 1. CREATE TABLE syntax will not be supported so existing BeamSQL pipelines 
> which has CREATE TABLE require changes.
> 2. It's required to type tedious EXTERNAL keyword every time, especially in 
> SQL Shell.
>  
> If we make the EXTERNAL keyword optional, we will have reversed pros and cons 
> above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Remove accidentally added .DS_Store file (#6275)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 94965de  Remove accidentally added .DS_Store file (#6275)
94965de is described below

commit 94965de9b9744b1e1aec1dc809a9aad597632256
Author: Thomas Weise 
AuthorDate: Fri Aug 24 13:02:27 2018 -0700

Remove accidentally added .DS_Store file (#6275)
---
 .gitignore   |   1 +
 .../org/apache/beam/runners/flink/translation/.DS_Store  | Bin 6148 -> 0 bytes
 2 files changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 8868b6e..f7e691a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -72,6 +72,7 @@ sdks/python/nosetests.xml
 
 # Ignore MacOSX files.
 **/.DS_Store/**/*
+**/.DS_Store
 
 # Ignore Jupyter notebook checkpoints.
 **/.ipynb_checkpoints/**/*
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/.DS_Store
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/.DS_Store
deleted file mode 100644
index 454b32d..000
Binary files 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/.DS_Store
 and /dev/null differ

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #896

[Apache Jenkins Server]

See 


Changes:

[github] Remove accidentally added .DS_Store file (#6275)

--
[...truncated 75.59 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.2.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  Could not find a version that satisfies the requirement six (from 
pyhamcrest->-r postcommit_requirements.txt (line 1)) (from versions: )
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
No matching distribution found for six (from pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ERROR
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting pyhamcrest (from -r postcommit_requireme

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137953&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137953
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:30
Start Date: 24/Aug/18 20:30
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415828302
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137953)
Time Spent: 1h 40m  (was: 1.5h)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137954&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137954
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:30
Start Date: 24/Aug/18 20:30
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415829415
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137954)
Time Spent: 1h 50m  (was: 1h 40m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137960&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137960
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415858719
 
 
   Run Java Precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137960)
Time Spent: 2h 50m  (was: 2h 40m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137955&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137955
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415841294
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137955)
Time Spent: 2h  (was: 1h 50m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137958&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137958
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415851835
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137958)
Time Spent: 2.5h  (was: 2h 20m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137956&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137956
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415842451
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137956)
Time Spent: 2h 10m  (was: 2h)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137961&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137961
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #6274: [BEAM-5035] Parallel 
builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415874446
 
 
   R: @lukecwik @Ardagan 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137961)
Time Spent: 3h  (was: 2h 50m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5035?focusedWorklogId=137957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137957
 ]

ASF GitHub Bot logged work on BEAM-5035:


Author: ASF GitHub Bot
Created on: 24/Aug/18 20:31
Start Date: 24/Aug/18 20:31
Worklog Time Spent: 10m 
  Work Description: apilloud removed a comment on issue #6274: [BEAM-5035] 
Parallel builds are very flaky, disable
URL: https://github.com/apache/beam/pull/6274#issuecomment-415851774
 
 
   run seed job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137957)
Time Spent: 2h 20m  (was: 2h 10m)

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Alan Myrvold
>Priority: Critical
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)