[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309400 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 10/Sep/19 00:21 Start Date: 10/Sep/19 00:21 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309400) Time Spent: 8.5h (was: 8h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 8.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309355
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 09/Sep/19 23:21
Start Date: 09/Sep/19 23:21
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r322494283
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import
org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.flink.api.java.ExecutionEnvironment;
+
+/** Singleton class that contains one {@link MultiInstanceFactory} per job. */
+public class FlinkExecutableStageContextFactory implements
ExecutableStageContext.Factory {
+
+ private static final FlinkExecutableStageContextFactory instance =
+ new FlinkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have
its own
+ // classloader and therefore its own instance of
FlinkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible
implementation of
+ // Factory.get(JobInfo), which in theory could be called with different
JobInfos.
+ private static final ConcurrentMap
jobFactories =
+ new ConcurrentHashMap<>();
+
+ private FlinkExecutableStageContextFactory() {}
+
+ public static FlinkExecutableStageContextFactory getInstance() {
+return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+MultiInstanceFactory state =
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309355)
Time Spent: 8h 10m (was: 8h)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309356
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 09/Sep/19 23:21
Start Date: 09/Sep/19 23:21
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r322494298
##
File path:
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import
org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+
+/**
+ * Singleton class that contains one {@link MultiInstanceFactory} per job.
Assumes it is safe to
+ * release the backing environment asynchronously.
+ */
+public class SparkExecutableStageContextFactory implements
ExecutableStageContext.Factory {
+
+ private static final SparkExecutableStageContextFactory instance =
+ new SparkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have
its own
+ // classloader and therefore its own instance of
SparkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible
implementation of
+ // Factory.get(JobInfo), which in theory could be called with different
JobInfos.
+ private static final ConcurrentMap
jobFactories =
+ new ConcurrentHashMap<>();
+
+ private SparkExecutableStageContextFactory() {}
+
+ public static SparkExecutableStageContextFactory getInstance() {
+return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+MultiInstanceFactory state =
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309356)
Time Spent: 8h 20m (was: 8h 10m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309344
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 09/Sep/19 22:55
Start Date: 09/Sep/19 22:55
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r322488140
##
File path:
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import
org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+
+/**
+ * Singleton class that contains one {@link MultiInstanceFactory} per job.
Assumes it is safe to
+ * release the backing environment asynchronously.
+ */
+public class SparkExecutableStageContextFactory implements
ExecutableStageContext.Factory {
+
+ private static final SparkExecutableStageContextFactory instance =
+ new SparkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have
its own
+ // classloader and therefore its own instance of
SparkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible
implementation of
+ // Factory.get(JobInfo), which in theory could be called with different
JobInfos.
+ private static final ConcurrentMap
jobFactories =
+ new ConcurrentHashMap<>();
+
+ private SparkExecutableStageContextFactory() {}
+
+ public static SparkExecutableStageContextFactory getInstance() {
+return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+MultiInstanceFactory state =
Review comment:
Same
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309344)
Time Spent: 8h (was: 7h 50m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309343
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 09/Sep/19 22:55
Start Date: 09/Sep/19 22:55
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r322483149
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import
org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.flink.api.java.ExecutionEnvironment;
+
+/** Singleton class that contains one {@link MultiInstanceFactory} per job. */
+public class FlinkExecutableStageContextFactory implements
ExecutableStageContext.Factory {
+
+ private static final FlinkExecutableStageContextFactory instance =
+ new FlinkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have
its own
+ // classloader and therefore its own instance of
FlinkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible
implementation of
+ // Factory.get(JobInfo), which in theory could be called with different
JobInfos.
+ private static final ConcurrentMap
jobFactories =
+ new ConcurrentHashMap<>();
+
+ private FlinkExecutableStageContextFactory() {}
+
+ public static FlinkExecutableStageContextFactory getInstance() {
+return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+MultiInstanceFactory state =
Review comment:
Shall we call this `jobFactory`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 309343)
Time Spent: 7h 50m (was: 7h 40m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309321 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 09/Sep/19 22:31 Start Date: 09/Sep/19 22:31 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-529692480 Run Python Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309321) Time Spent: 7.5h (was: 7h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 7.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309322 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 09/Sep/19 22:31 Start Date: 09/Sep/19 22:31 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-529692533 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309322) Time Spent: 7h 40m (was: 7.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 7h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309181 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 09/Sep/19 20:13 Start Date: 09/Sep/19 20:13 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-529646438 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309181) Time Spent: 7h 20m (was: 7h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 7h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=309180=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-309180 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 09/Sep/19 20:13 Start Date: 09/Sep/19 20:13 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-529646382 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 309180) Time Spent: 7h 10m (was: 7h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 7h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=308202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-308202 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 06/Sep/19 22:45 Start Date: 06/Sep/19 22:45 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-529038195 @angoenka I have refactored this quite a bit. Unfortunately I only managed to eliminate a small amount of the complexity as most of it seems necessary. PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 308202) Time Spent: 7h (was: 6h 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 7h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=307543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307543
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 23:57
Start Date: 05/Sep/19 23:57
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321531229
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
##
@@ -102,7 +100,8 @@ private JobFactoryState(int maxFactories) {
new ConcurrentHashMap<>();
@Override
-public FlinkExecutableStageContext get(JobInfo jobInfo) {
+public ExecutableStageContext get(
+JobInfo jobInfo, SerializableFunction
isReleaseSynchronous) {
Review comment:
As discussed offline, I made separate contexts for Flink and Spark.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307543)
Time Spent: 6h 50m (was: 6h 40m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=307435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307435
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 21:02
Start Date: 05/Sep/19 21:02
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321478397
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
##
@@ -102,7 +100,8 @@ private JobFactoryState(int maxFactories) {
new ConcurrentHashMap<>();
@Override
-public FlinkExecutableStageContext get(JobInfo jobInfo) {
+public ExecutableStageContext get(
+JobInfo jobInfo, SerializableFunction
isReleaseSynchronous) {
Review comment:
I still feel that we should not have `isReleaseSynchronous` in `get` method
as the usage can be unpredictable in case where we call `get` method at 2
locations for the same jobInfo but with different `isReleaseSynchronous`
method. As we are using caching, the 1st `isReleaseSynchronous` will be applied
and can lead to concurrency bug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307435)
Time Spent: 6h 40m (was: 6.5h)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=307346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307346
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 17:17
Start Date: 05/Sep/19 17:17
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321384103
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java
##
@@ -26,42 +26,46 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.sdk.function.ThrowingFunction;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
-import org.apache.flink.annotation.VisibleForTesting;
-import org.apache.flink.api.java.ExecutionEnvironment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * {@link FlinkExecutableStageContext.Factory} which counts
FlinkExecutableStageContext reference
- * for book keeping.
+ * {@link ExecutableStageContext.Factory} which counts ExecutableStageContext
reference for book
+ * keeping.
*/
-public class ReferenceCountingFlinkExecutableStageContextFactory
-implements FlinkExecutableStageContext.Factory {
+public class ReferenceCountingExecutableStageContextFactory
+implements ExecutableStageContext.Factory {
private static final Logger LOG =
-
LoggerFactory.getLogger(ReferenceCountingFlinkExecutableStageContextFactory.class);
+
LoggerFactory.getLogger(ReferenceCountingExecutableStageContextFactory.class);
private static final int MAX_RETRY = 3;
private final Creator creator;
private transient volatile ScheduledExecutorService executor;
private transient volatile ConcurrentHashMap
keyRegistry;
+ private final SerializableFunction isReleaseSynchronous;
- public static ReferenceCountingFlinkExecutableStageContextFactory
create(Creator creator) {
-return new ReferenceCountingFlinkExecutableStageContextFactory(creator);
+ public static ReferenceCountingExecutableStageContextFactory create(
+ Creator creator, SerializableFunction
isReleaseSynchronous) {
+return new ReferenceCountingExecutableStageContextFactory(creator,
isReleaseSynchronous);
}
- private ReferenceCountingFlinkExecutableStageContextFactory(Creator creator)
{
+ private ReferenceCountingExecutableStageContextFactory(
+ Creator creator, SerializableFunction
isReleaseSynchronous) {
this.creator = creator;
+this.isReleaseSynchronous = isReleaseSynchronous;
}
@Override
- public FlinkExecutableStageContext get(JobInfo jobInfo) {
+ public ExecutableStageContext get(
+ JobInfo jobInfo, SerializableFunction
isReleaseSynchronous) {
Review comment:
Refactored to avoid this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307346)
Time Spent: 6.5h (was: 6h 20m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
>
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=307345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-307345
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 17:16
Start Date: 05/Sep/19 17:16
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321383923
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
##
@@ -116,7 +116,7 @@ public FlinkExecutableStageContext get(JobInfo jobInfo) {
.intValue());
});
- return state.getFactory().get(jobInfo);
+ return state.getFactory(isReleaseSynchronous).get(jobInfo,
isReleaseSynchronous);
Review comment:
Refactored to avoid this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 307345)
Time Spent: 6h 20m (was: 6h 10m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=306799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306799
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 00:44
Start Date: 05/Sep/19 00:44
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321029943
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java
##
@@ -26,42 +26,46 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.sdk.function.ThrowingFunction;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
-import org.apache.flink.annotation.VisibleForTesting;
-import org.apache.flink.api.java.ExecutionEnvironment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * {@link FlinkExecutableStageContext.Factory} which counts
FlinkExecutableStageContext reference
- * for book keeping.
+ * {@link ExecutableStageContext.Factory} which counts ExecutableStageContext
reference for book
+ * keeping.
*/
-public class ReferenceCountingFlinkExecutableStageContextFactory
-implements FlinkExecutableStageContext.Factory {
+public class ReferenceCountingExecutableStageContextFactory
+implements ExecutableStageContext.Factory {
private static final Logger LOG =
-
LoggerFactory.getLogger(ReferenceCountingFlinkExecutableStageContextFactory.class);
+
LoggerFactory.getLogger(ReferenceCountingExecutableStageContextFactory.class);
private static final int MAX_RETRY = 3;
private final Creator creator;
private transient volatile ScheduledExecutorService executor;
private transient volatile ConcurrentHashMap
keyRegistry;
+ private final SerializableFunction isReleaseSynchronous;
- public static ReferenceCountingFlinkExecutableStageContextFactory
create(Creator creator) {
-return new ReferenceCountingFlinkExecutableStageContextFactory(creator);
+ public static ReferenceCountingExecutableStageContextFactory create(
+ Creator creator, SerializableFunction
isReleaseSynchronous) {
+return new ReferenceCountingExecutableStageContextFactory(creator,
isReleaseSynchronous);
}
- private ReferenceCountingFlinkExecutableStageContextFactory(Creator creator)
{
+ private ReferenceCountingExecutableStageContextFactory(
+ Creator creator, SerializableFunction
isReleaseSynchronous) {
this.creator = creator;
+this.isReleaseSynchronous = isReleaseSynchronous;
}
@Override
- public FlinkExecutableStageContext get(JobInfo jobInfo) {
+ public ExecutableStageContext get(
+ JobInfo jobInfo, SerializableFunction
isReleaseSynchronous) {
Review comment:
`isReleaseSynchronous` is not used
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 306799)
Time Spent: 6h 10m (was: 6h)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
>
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[
https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=306798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-306798
]
ASF GitHub Bot logged work on BEAM-7600:
Author: ASF GitHub Bot
Created on: 05/Sep/19 00:44
Start Date: 05/Sep/19 00:44
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9095: [BEAM-7600]
borrow SDK harness management code into Spark runner
URL: https://github.com/apache/beam/pull/9095#discussion_r321033879
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
##
@@ -116,7 +116,7 @@ public FlinkExecutableStageContext get(JobInfo jobInfo) {
.intValue());
});
- return state.getFactory().get(jobInfo);
+ return state.getFactory(isReleaseSynchronous).get(jobInfo,
isReleaseSynchronous);
Review comment:
Same, I think we should pass `isReleaseSynchronous` only at 1 place to avoid
confusion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 306798)
Time Spent: 6h (was: 5h 50m)
> Spark portable runner: reuse SDK harness
>
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
> Labels: portability-spark
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is
> run [1], which is expensive. We should be able to re-use code from the Flink
> runner to re-use the SDK harness [2].
>
> [1]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2]
> [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=302462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302462 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 27/Aug/19 23:33 Start Date: 27/Aug/19 23:33 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-525522937 @angoenka PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302462) Time Spent: 5h 50m (was: 5h 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5h 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=302215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-302215 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 27/Aug/19 16:56 Start Date: 27/Aug/19 16:56 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-525390592 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 302215) Time Spent: 5h 40m (was: 5.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=300514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300514 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 23/Aug/19 21:24 Start Date: 23/Aug/19 21:24 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-524467042 Alright, I've finally ironed out the problems with this one. However, we should merge #9410 first for best results. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300514) Time Spent: 5.5h (was: 5h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=299045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-299045 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 21/Aug/19 22:21 Start Date: 21/Aug/19 22:21 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-523672131 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 299045) Time Spent: 5h 20m (was: 5h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=289750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289750 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 06/Aug/19 14:42 Start Date: 06/Aug/19 14:42 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-518700148 Is this change ready for review? If so, gentle reminder @angoenka. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289750) Time Spent: 5h 10m (was: 5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279473 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906985 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279473) Time Spent: 5h (was: 4h 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279472 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906102 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279472) Time Spent: 4h 50m (was: 4h 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4h 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279469 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512905978 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279469) Time Spent: 4h 20m (was: 4h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279470 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906021 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279470) Time Spent: 4.5h (was: 4h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279468 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512902060 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279468) Time Spent: 4h 10m (was: 4h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279471 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:15 Start Date: 19/Jul/19 03:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906072 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279471) Time Spent: 4h 40m (was: 4.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279459 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-513074048 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279459) Time Spent: 2h 50m (was: 2h 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2h 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279467 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512902012 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279467) Time Spent: 4h (was: 3h 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 4h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279464=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279464 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895508 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279464) Time Spent: 3.5h (was: 3h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279465=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279465 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512901881 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279465) Time Spent: 3h 40m (was: 3.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279466 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512901983 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279466) Time Spent: 3h 50m (was: 3h 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3h 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279461 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895062 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279461) Time Spent: 3h (was: 2h 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279462 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895099 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279462) Time Spent: 3h 10m (was: 3h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279463 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 19/Jul/19 03:14 Start Date: 19/Jul/19 03:14 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895568 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279463) Time Spent: 3h 20m (was: 3h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 3h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279389 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 23:02 Start Date: 18/Jul/19 23:02 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-513025131 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279389) Time Spent: 2h 40m (was: 2.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279162 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:15 Start Date: 18/Jul/19 17:15 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906985 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279162) Time Spent: 2.5h (was: 2h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279155 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:12 Start Date: 18/Jul/19 17:12 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906021 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279155) Time Spent: 2h (was: 1h 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279158 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:12 Start Date: 18/Jul/19 17:12 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906102 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279158) Time Spent: 2h 20m (was: 2h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279154 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:12 Start Date: 18/Jul/19 17:12 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512905978 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279154) Time Spent: 1h 50m (was: 1h 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279156 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:12 Start Date: 18/Jul/19 17:12 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512906072 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279156) Time Spent: 2h 10m (was: 2h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 2h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279146 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:00 Start Date: 18/Jul/19 17:00 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512901983 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279146) Time Spent: 1.5h (was: 1h 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279144 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:00 Start Date: 18/Jul/19 17:00 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512901881 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279144) Time Spent: 1h 10m (was: 1h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 10m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279147 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:00 Start Date: 18/Jul/19 17:00 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512902060 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279147) Time Spent: 1h 40m (was: 1.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279145 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 17:00 Start Date: 18/Jul/19 17:00 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512902012 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279145) Time Spent: 1h 20m (was: 1h 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279127=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279127 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 16:44 Start Date: 18/Jul/19 16:44 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512896013 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279127) Time Spent: 1h (was: 50m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 1h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279126 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 16:43 Start Date: 18/Jul/19 16:43 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895568 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279126) Time Spent: 50m (was: 40m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 50m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279125=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279125 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 16:43 Start Date: 18/Jul/19 16:43 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895508 Run Java Spark PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279125) Time Spent: 40m (was: 0.5h) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 40m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279122 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 16:42 Start Date: 18/Jul/19 16:42 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895062 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279122) Time Spent: 20m (was: 10m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 20m > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=279123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279123 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 16:42 Start Date: 18/Jul/19 16:42 Worklog Time Spent: 10m Work Description: ibzib commented on issue #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095#issuecomment-512895099 Run Python_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279123) Time Spent: 0.5h (was: 20m) > Spark portable runner: reuse SDK harness > > > Key: BEAM-7600 > URL: https://issues.apache.org/jira/browse/BEAM-7600 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > Labels: portability-spark > Time Spent: 0.5h > Remaining Estimate: 0h > > Right now, we're creating a new SDK harness every time an executable stage is > run [1], which is expensive. We should be able to re-use code from the Flink > runner to re-use the SDK harness [2]. > > [1] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135] > [2] > [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java] -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7600) Spark portable runner: reuse SDK harness
[ https://issues.apache.org/jira/browse/BEAM-7600?focusedWorklogId=278673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-278673 ] ASF GitHub Bot logged work on BEAM-7600: Author: ASF GitHub Bot Created on: 18/Jul/19 02:57 Start Date: 18/Jul/19 02:57 Worklog Time Spent: 10m Work Description: ibzib commented on pull request #9095: [BEAM-7600] borrow SDK harness management code into Spark runner URL: https://github.com/apache/beam/pull/9095 Now the Spark runner can reuse SDK harnesses, and multiple SDK harness can be used. The latter will hopefully enable multicore processing on TFX, for example. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) Pre-Commit Tests
