[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=154120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154120
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 13/Oct/18 18:40
Start Date: 13/Oct/18 18:40
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224968563
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1498,6 +1498,7 @@ artifactId=${project.name}
testClassesDirs =
project.files(project.project(":beam-sdks-java-core").sourceSets.test.output.classesDirs,
project.project(":beam-runners-core-java").sourceSets.test.output.classesDirs)
maxParallelForks config.parallelism
useJUnit(config.testCategories)
+dependsOn ':beam-sdks-java-container:docker'
Review comment:
Is this later going to change to use the process environment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 154120)
Time Spent: 33h 40m (was: 33.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 33h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153992&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153992
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 12/Oct/18 18:57
Start Date: 12/Oct/18 18:57
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224885162
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
You can base a PR on another PR's branch. Not sure how much sense that makes
because then you would merge into your fork instead of the upstream repo. So I
take that back :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153992)
Time Spent: 33.5h (was: 33h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 33.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153972&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153972
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 12/Oct/18 18:13
Start Date: 12/Oct/18 18:13
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224873292
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
Maybe I am missing some thing.
For this PR, I based my branch java_pvr_jenkins on
java_pvr_cache_environments and then created the PR (merge 1 commit into
apache:master from angoenka:java_pvr_jenkins).
This picks the commits from both branches java_pvr_jenkins and
java_pvr_cache_environments.
Is there anyway to avoid picking commits from the base PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153972)
Time Spent: 33h 20m (was: 33h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 33h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153971&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153971
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 12/Oct/18 18:12
Start Date: 12/Oct/18 18:12
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224873292
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
Maybe I am missing some thing.try t
For this PR, I based my branch java_pvr_jenkins on
java_pvr_cache_environments and then created the PR (merge 1 commit into
apache:master from angoenka:java_pvr_jenkins).
This picks the commits from both branches java_pvr_jenkins and
java_pvr_cache_environments.
Is there anyway to avoid picking commits from the base PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153971)
Time Spent: 33h 10m (was: 33h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 33h 10m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153840
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 12/Oct/18 09:04
Start Date: 12/Oct/18 09:04
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224718474
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
> There is no way to make PR depend upon other PR in git so had to resort to
adding 2 separate commits.
Not true, you can just specify the branch of the other PR as base branch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153840)
Time Spent: 33h (was: 32h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 33h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153764
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 12/Oct/18 05:10
Start Date: 12/Oct/18 05:10
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224672134
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
Yes, that's one of those situations where 2 separate commits in the same PR
make sense. But since you have the other PR under review already, why not wrap
that up first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153764)
Time Spent: 32h 50m (was: 32h 40m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 32h 50m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153740&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153740 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 11/Oct/18 23:42 Start Date: 11/Oct/18 23:42 Worklog Time Spent: 10m Work Description: angoenka commented on a change in pull request #6592: [BEAM-4176] Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#discussion_r224598766 ## File path: .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink.groovy ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import CommonJobProperties as commonJobProperties Review comment: I checked, its highlighted in other groovy files as well. It might have something to do with groovy files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153740) Time Spent: 32.5h (was: 32h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 32.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153741&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153741
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 11/Oct/18 23:42
Start Date: 11/Oct/18 23:42
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224601348
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
I had to incorporate those change to make this PR work.
There is no way to make PR depend upon other PR in git so had to resort to
adding 2 separate commits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153741)
Time Spent: 32h 40m (was: 32.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 32h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153450 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 11/Oct/18 09:40 Start Date: 11/Oct/18 09:40 Worklog Time Spent: 10m Work Description: mxm commented on a change in pull request #6592: [BEAM-4176] Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#discussion_r224374813 ## File path: .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink.groovy ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import CommonJobProperties as commonJobProperties Review comment: Not sure why GitHub highlights this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153450) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 32h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153449&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153449
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 11/Oct/18 09:40
Start Date: 11/Oct/18 09:40
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6592:
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224381088
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
##
@@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
WrappedContext wrapper = getCache().get(jobInfo.jobId());
Preconditions.checkState(
wrapper != null, "Releasing context for unknown job: " +
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to
the classloader not being
-// available anymore after the tasks have been removed from the execution
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+ // Schedule task to clean the container later.
+ // Ensure that this class is loaded in the parent Flink classloader.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis,
TimeUnit.MILLISECONDS);
Review comment:
These changes are already contained in #6638. You could base this PR on
#6638.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 153449)
Time Spent: 32h 20m (was: 32h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 32h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153294&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153294 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 10/Oct/18 21:38 Start Date: 10/Oct/18 21:38 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-428742000 The PR can be reviewed now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153294) Time Spent: 32h 10m (was: 32h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 32h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153293 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 10/Oct/18 21:36 Start Date: 10/Oct/18 21:36 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-428741134 Its not ideal but we are able to finish in 45 min while a native flink Validates Runner takes ~20 min Successful run https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_PR/3/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153293) Time Spent: 32h (was: 31h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 32h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153280 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 10/Oct/18 20:50 Start Date: 10/Oct/18 20:50 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-428727017 Run Java Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153280) Time Spent: 31h 50m (was: 31h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153277&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153277 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 10/Oct/18 20:46 Start Date: 10/Oct/18 20:46 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-428725651 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153277) Time Spent: 31h 40m (was: 31.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152424&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152424 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 08/Oct/18 21:45 Start Date: 08/Oct/18 21:45 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-427989489 Tests are passing locally but they are timing out after 100min on jenkins. I think we need to reuse artifact staging to speed it up. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 152424) Time Spent: 31.5h (was: 31h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152383 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 08/Oct/18 20:05 Start Date: 08/Oct/18 20:05 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-427962144 R: @mxm @tweise @robertwb @ryan-williams This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 152383) Time Spent: 31h 20m (was: 31h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152381 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 08/Oct/18 19:59 Start Date: 08/Oct/18 19:59 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-427960658 Run Java Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 152381) Time Spent: 31h 10m (was: 31h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152369 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 08/Oct/18 19:42 Start Date: 08/Oct/18 19:42 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-427955852 Run Java Flink PortableValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 152369) Time Spent: 31h (was: 30h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 31h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152365 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 08/Oct/18 19:32 Start Date: 08/Oct/18 19:32 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592#issuecomment-427953217 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 152365) Time Spent: 30h 50m (was: 30h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 30h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=152187&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-152187
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 08/Oct/18 10:25
Start Date: 08/Oct/18 10:25
Worklog Time Spent: 10m
Work Description: mxm closed pull request #6563: [BEAM-4176] Cleanup SDK
Harness docker container
URL: https://github.com/apache/beam/pull/6563
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
index 6a8988e76fa..4c3d5bf690b 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
@@ -28,6 +28,7 @@
import java.util.Map;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
+import
org.apache.beam.vendor.protobuf.v3.com.google.protobuf.InvalidProtocolBufferException;
import org.apache.beam.vendor.protobuf.v3.com.google.protobuf.Struct;
import org.apache.beam.vendor.protobuf.v3.com.google.protobuf.util.JsonFormat;
@@ -66,38 +67,50 @@ public static Struct toProto(PipelineOptions options) {
JsonFormat.parser().merge(MAPPER.writeValueAsString(optionsUsingUrns),
builder);
return builder.build();
} catch (IOException e) {
- throw new RuntimeException(e);
+ throw new RuntimeException("Failed to convert PipelineOptions to
Protocol", e);
}
}
/** Converts the provided {@link Struct} into {@link PipelineOptions}. */
- public static PipelineOptions fromProto(Struct protoOptions) throws
IOException {
-Map mapWithoutUrns = new HashMap<>();
-TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
-Iterator optionsKeys = rootOptions.fieldNames();
-while (optionsKeys.hasNext()) {
- String optionKey = optionsKeys.next();
- TreeNode optionValue = rootOptions.get(optionKey);
- mapWithoutUrns.put(
- CaseFormat.LOWER_UNDERSCORE.to(
- CaseFormat.LOWER_CAMEL,
- optionKey.substring("beam:option:".length(), optionKey.length()
- ":v1".length())),
- optionValue);
+ public static PipelineOptions fromProto(Struct protoOptions) {
+try {
+ Map mapWithoutUrns = new HashMap<>();
+ TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
+ Iterator optionsKeys = rootOptions.fieldNames();
+ while (optionsKeys.hasNext()) {
+String optionKey = optionsKeys.next();
+TreeNode optionValue = rootOptions.get(optionKey);
+mapWithoutUrns.put(
+CaseFormat.LOWER_UNDERSCORE.to(
+CaseFormat.LOWER_CAMEL,
+optionKey.substring("beam:option:".length(),
optionKey.length() - ":v1".length())),
+optionValue);
+ }
+ return MAPPER.readValue(
+ MAPPER.writeValueAsString(ImmutableMap.of("options",
mapWithoutUrns)),
+ PipelineOptions.class);
+} catch (IOException e) {
+ throw new RuntimeException("Failed to read PipelineOptions from
Protocol", e);
}
-return MAPPER.readValue(
-MAPPER.writeValueAsString(ImmutableMap.of("options", mapWithoutUrns)),
-PipelineOptions.class);
}
/** Converts the provided Json{@link String} into {@link PipelineOptions}. */
- public static PipelineOptions fromJson(String optionsJson) throws
IOException {
-Struct.Builder builder = Struct.newBuilder();
-JsonFormat.parser().merge(optionsJson, builder);
-return fromProto(builder.build());
+ public static PipelineOptions fromJson(String optionsJson) {
+try {
+ Struct.Builder builder = Struct.newBuilder();
+ JsonFormat.parser().merge(optionsJson, builder);
+ return fromProto(builder.build());
+} catch (IOException e) {
+ throw new RuntimeException("Failed to read PipelineOptions from JSON",
e);
+}
}
/** Converts the provided {@link PipelineOptions} into Json{@link String}. */
- public static String toJson(PipelineOptions options) throws IOException {
-return JsonFormat.printer().print(toProto(options));
+ public static String toJson(PipelineOptions options) {
+try {
+ return JsonFormat.printer().print(toProto(options));
+} catch (InvalidProtocolBuf
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151871 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 05/Oct/18 21:08 Start Date: 05/Oct/18 21:08 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-427499210 Thanks @mxm Update the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151871) Time Spent: 30.5h (was: 30h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 30.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151868&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151868 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 05/Oct/18 21:07 Start Date: 05/Oct/18 21:07 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6592: [BEAM-4176] WIP Enable Post Commit JAVA PVR tests for Flink URL: https://github.com/apache/beam/pull/6592 **Please** add a meaningful description for your change here Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151868) Time Spent: 30h 20m (was: 30h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam >
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151847
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 20:18
Start Date: 05/Oct/18 20:18
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6534: [BEAM-4176] Filtering
Metrics and Splittable ParDo tests for Portable Runner
URL: https://github.com/apache/beam/pull/6534
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 5b4154d957d..076ad88bdff 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -209,12 +209,21 @@ class BeamModulePlugin implements Plugin {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesAttemptedMetrics'
excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCounterMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesCustomWindowMerging'
excludeCategories 'org.apache.beam.sdk.testing.UsesDistributionMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesAttemptedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesTimersInParDo'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesFailureMessage'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesStatefulParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+ excludeCategories 'org.apache.beam.sdk.testing.UsesTimersInParDo'
+ //SplitableDoFnTests
+ excludeCategories
'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
+ excludeCategories
'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
+ excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
}
// Configuration for the classpath when running the test.
Configuration testClasspathConfiguration
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java
new file mode 100644
index 000..85d62909f2e
--- /dev/null
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/**
+ * Category tag for tests which validate that currect failure message is
provided by failed
+ * pipeline.
+ */
+public interface UsesFailureMessage extends NeedsRunner {}
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
index c4aa9cae103..bedbb5da4e9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
@@ -335,7 +335,7 @@ public void testNotEqualTo() throws Exception {
/** Test that we throw an error for false assertion on singleton. */
@Test
- @Category(ValidatesRunner.class)
+ @Category({ValidatesRunner.class, UsesFailureMessage.class})
public void testPAssert
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151827&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151827 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 05/Oct/18 19:50 Start Date: 05/Oct/18 19:50 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6534: [BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner URL: https://github.com/apache/beam/pull/6534#issuecomment-427479812 Ping! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151827) Time Spent: 30h (was: 29h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 30h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151825&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151825
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 19:48
Start Date: 05/Oct/18 19:48
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r223119729
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
##
@@ -225,6 +235,11 @@ private static ServerFactory getServerFactory() {
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+private final boolean retain;
Review comment:
Sure!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151825)
Time Spent: 29h 50m (was: 29h 40m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29h 50m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151563
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 09:50
Start Date: 05/Oct/18 09:50
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222949495
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
##
@@ -225,6 +235,11 @@ private static ServerFactory getServerFactory() {
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+private final boolean retain;
Review comment:
Would rename this to `retainDockerContainer`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151563)
Time Spent: 29h 20m (was: 29h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151565
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 09:50
Start Date: 05/Oct/18 09:50
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222950138
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
##
@@ -71,33 +72,45 @@ public static Struct toProto(PipelineOptions options) {
}
/** Converts the provided {@link Struct} into {@link PipelineOptions}. */
- public static PipelineOptions fromProto(Struct protoOptions) throws
IOException {
-Map mapWithoutUrns = new HashMap<>();
-TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
-Iterator optionsKeys = rootOptions.fieldNames();
-while (optionsKeys.hasNext()) {
- String optionKey = optionsKeys.next();
- TreeNode optionValue = rootOptions.get(optionKey);
- mapWithoutUrns.put(
- CaseFormat.LOWER_UNDERSCORE.to(
- CaseFormat.LOWER_CAMEL,
- optionKey.substring("beam:option:".length(), optionKey.length()
- ":v1".length())),
- optionValue);
+ public static PipelineOptions fromProto(Struct protoOptions) {
+try {
+ Map mapWithoutUrns = new HashMap<>();
+ TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
+ Iterator optionsKeys = rootOptions.fieldNames();
+ while (optionsKeys.hasNext()) {
+String optionKey = optionsKeys.next();
+TreeNode optionValue = rootOptions.get(optionKey);
+mapWithoutUrns.put(
+CaseFormat.LOWER_UNDERSCORE.to(
+CaseFormat.LOWER_CAMEL,
+optionKey.substring("beam:option:".length(),
optionKey.length() - ":v1".length())),
+optionValue);
+ }
+ return MAPPER.readValue(
+ MAPPER.writeValueAsString(ImmutableMap.of("options",
mapWithoutUrns)),
+ PipelineOptions.class);
+} catch (IOException e) {
+ throw new RuntimeException(e);
}
-return MAPPER.readValue(
-MAPPER.writeValueAsString(ImmutableMap.of("options", mapWithoutUrns)),
-PipelineOptions.class);
}
/** Converts the provided Json{@link String} into {@link PipelineOptions}. */
- public static PipelineOptions fromJson(String optionsJson) throws
IOException {
-Struct.Builder builder = Struct.newBuilder();
-JsonFormat.parser().merge(optionsJson, builder);
-return fromProto(builder.build());
+ public static PipelineOptions fromJson(String optionsJson) {
+try {
+ Struct.Builder builder = Struct.newBuilder();
+ JsonFormat.parser().merge(optionsJson, builder);
+ return fromProto(builder.build());
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
}
/** Converts the provided {@link PipelineOptions} into Json{@link String}. */
- public static String toJson(PipelineOptions options) throws IOException {
-return JsonFormat.printer().print(toProto(options));
+ public static String toJson(PipelineOptions options) {
+try {
+ return JsonFormat.printer().print(toProto(options));
+} catch (InvalidProtocolBufferException e) {
+ throw new RuntimeException(e);
Review comment:
I find it nice to add a descriptive message, e.g.
```new RuntimeException("Failed to convert PipelineOptions to JSON", e);```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151565)
Time Spent: 29.5h (was: 29h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This messa
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151566&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151566
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 09:50
Start Date: 05/Oct/18 09:50
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222950625
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
##
@@ -71,33 +72,45 @@ public static Struct toProto(PipelineOptions options) {
}
/** Converts the provided {@link Struct} into {@link PipelineOptions}. */
- public static PipelineOptions fromProto(Struct protoOptions) throws
IOException {
-Map mapWithoutUrns = new HashMap<>();
-TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
-Iterator optionsKeys = rootOptions.fieldNames();
-while (optionsKeys.hasNext()) {
- String optionKey = optionsKeys.next();
- TreeNode optionValue = rootOptions.get(optionKey);
- mapWithoutUrns.put(
- CaseFormat.LOWER_UNDERSCORE.to(
- CaseFormat.LOWER_CAMEL,
- optionKey.substring("beam:option:".length(), optionKey.length()
- ":v1".length())),
- optionValue);
+ public static PipelineOptions fromProto(Struct protoOptions) {
+try {
+ Map mapWithoutUrns = new HashMap<>();
+ TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
+ Iterator optionsKeys = rootOptions.fieldNames();
+ while (optionsKeys.hasNext()) {
+String optionKey = optionsKeys.next();
+TreeNode optionValue = rootOptions.get(optionKey);
+mapWithoutUrns.put(
+CaseFormat.LOWER_UNDERSCORE.to(
+CaseFormat.LOWER_CAMEL,
+optionKey.substring("beam:option:".length(),
optionKey.length() - ":v1".length())),
+optionValue);
+ }
+ return MAPPER.readValue(
+ MAPPER.writeValueAsString(ImmutableMap.of("options",
mapWithoutUrns)),
+ PipelineOptions.class);
+} catch (IOException e) {
+ throw new RuntimeException(e);
}
-return MAPPER.readValue(
-MAPPER.writeValueAsString(ImmutableMap.of("options", mapWithoutUrns)),
-PipelineOptions.class);
}
/** Converts the provided Json{@link String} into {@link PipelineOptions}. */
- public static PipelineOptions fromJson(String optionsJson) throws
IOException {
-Struct.Builder builder = Struct.newBuilder();
-JsonFormat.parser().merge(optionsJson, builder);
-return fromProto(builder.build());
+ public static PipelineOptions fromJson(String optionsJson) {
+try {
+ Struct.Builder builder = Struct.newBuilder();
+ JsonFormat.parser().merge(optionsJson, builder);
+ return fromProto(builder.build());
+} catch (IOException e) {
+ throw new RuntimeException(e);
Review comment:
I find it nice to add a descriptive message, e.g.
```new RuntimeException("Failed to read PipelineOptions from JSON", e);```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151566)
Time Spent: 29h 40m (was: 29.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151564&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151564
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 05/Oct/18 09:50
Start Date: 05/Oct/18 09:50
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222951038
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
##
@@ -71,33 +72,45 @@ public static Struct toProto(PipelineOptions options) {
}
/** Converts the provided {@link Struct} into {@link PipelineOptions}. */
- public static PipelineOptions fromProto(Struct protoOptions) throws
IOException {
-Map mapWithoutUrns = new HashMap<>();
-TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
-Iterator optionsKeys = rootOptions.fieldNames();
-while (optionsKeys.hasNext()) {
- String optionKey = optionsKeys.next();
- TreeNode optionValue = rootOptions.get(optionKey);
- mapWithoutUrns.put(
- CaseFormat.LOWER_UNDERSCORE.to(
- CaseFormat.LOWER_CAMEL,
- optionKey.substring("beam:option:".length(), optionKey.length()
- ":v1".length())),
- optionValue);
+ public static PipelineOptions fromProto(Struct protoOptions) {
+try {
+ Map mapWithoutUrns = new HashMap<>();
+ TreeNode rootOptions =
MAPPER.readTree(JsonFormat.printer().print(protoOptions));
+ Iterator optionsKeys = rootOptions.fieldNames();
+ while (optionsKeys.hasNext()) {
+String optionKey = optionsKeys.next();
+TreeNode optionValue = rootOptions.get(optionKey);
+mapWithoutUrns.put(
+CaseFormat.LOWER_UNDERSCORE.to(
+CaseFormat.LOWER_CAMEL,
+optionKey.substring("beam:option:".length(),
optionKey.length() - ":v1".length())),
+optionValue);
+ }
+ return MAPPER.readValue(
+ MAPPER.writeValueAsString(ImmutableMap.of("options",
mapWithoutUrns)),
+ PipelineOptions.class);
+} catch (IOException e) {
+ throw new RuntimeException(e);
Review comment:
I find it nice to add a descriptive message, e.g.
```
new RuntimeException("Failed to read PipelineOptions from Protocol", e);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151564)
Time Spent: 29.5h (was: 29h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151472&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151472 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 05/Oct/18 01:23 Start Date: 05/Oct/18 01:23 Worklog Time Spent: 10m Work Description: tweise commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-427218323 @angoenka for future PRs, please tag the JIRA in the commit messages. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151472) Time Spent: 29h 10m (was: 29h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 29h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151346
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 20:01
Start Date: 04/Oct/18 20:01
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222808373
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions,
PipelineOptions {
+
+ @Description("Delete dynamically created SDK Harness containers.")
Review comment:
Makes sense. Based on herohde comment, I will update it to
`retainDockerContainers`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151346)
Time Spent: 29h (was: 28h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 29h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151345
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 20:00
Start Date: 04/Oct/18 20:00
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222807966
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
##
@@ -225,6 +233,11 @@ private static ServerFactory getServerFactory() {
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+private final boolean cleanup;
Review comment:
Makes sense. Will update the file name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151345)
Time Spent: 28h 50m (was: 28h 40m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h 50m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151257
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 16:09
Start Date: 04/Oct/18 16:09
Worklog Time Spent: 10m
Work Description: herohde commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222730928
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions,
PipelineOptions {
Review comment:
This option is more specific than portable jobs. It should rather be
ManualDockerEnvironmentOptions, say, and not extend other options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151257)
Time Spent: 28h 40m (was: 28.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151256
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 16:09
Start Date: 04/Oct/18 16:09
Worklog Time Spent: 10m
Work Description: herohde commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222733149
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
##
@@ -225,6 +233,11 @@ private static ServerFactory getServerFactory() {
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+private final boolean cleanup;
Review comment:
I suspect there will be many other docker options that we eventually would
want to expose, especially for local runs. Mounts, networking, etc. Perhaps it
would make sense to use a structure similar to ManualDockerEnvironmentOptions
mentioned below to avoid updating the plumbing multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151256)
Time Spent: 28h 40m (was: 28.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151120
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 10:39
Start Date: 04/Oct/18 10:39
Worklog Time Spent: 10m
Work Description: mxm closed pull request #6537: [BEAM-4176] Store and
serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
index 0318600a4d5..155977ca525 100644
---
a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
+++
b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
@@ -35,7 +35,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-class JobServicePipelineResult implements PipelineResult {
+class JobServicePipelineResult implements PipelineResult, AutoCloseable {
private static final long POLL_INTERVAL_MS = 10 * 1000;
@@ -43,14 +43,19 @@
private final ByteString jobId;
private final CloseableResource jobService;
+ @Nullable private State terminationState;
JobServicePipelineResult(ByteString jobId,
CloseableResource jobService) {
this.jobId = jobId;
this.jobService = jobService;
+this.terminationState = null;
}
@Override
public State getState() {
+if (terminationState != null) {
+ return terminationState;
+}
JobServiceBlockingStub stub = jobService.get();
GetJobStateResponse response =
stub.getState(GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build());
@@ -89,6 +94,9 @@ public State waitUntilFinish(Duration duration) {
@Override
public State waitUntilFinish() {
+if (terminationState != null) {
+ return terminationState;
+}
JobServiceBlockingStub stub = jobService.get();
GetJobStateRequest request =
GetJobStateRequest.newBuilder().setJobIdBytes(jobId).build();
GetJobStateResponse response = stub.getState(request);
@@ -103,11 +111,8 @@ public State waitUntilFinish() {
response = stub.getState(request);
lastState = getJavaState(response.getState());
}
-try {
- jobService.close();
-} catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
-}
+close();
+terminationState = lastState;
return lastState;
}
@@ -116,6 +121,14 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() {
+try (CloseableResource jobService =
this.jobService) {
+} catch (Exception e) {
+ LOG.warn("Error cleaning up job service", e);
+}
+ }
+
private static State getJavaState(JobApi.JobState.Enum protoState) {
switch (protoState) {
case UNSPECIFIED:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151120)
Time Spent: 28.5h (was: 28h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151026&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151026
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 04:22
Start Date: 04/Oct/18 04:22
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222534143
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java
##
@@ -29,23 +31,35 @@
import
org.apache.beam.runners.fnexecution.environment.EmbeddedEnvironmentFactory;
import
org.apache.beam.runners.fnexecution.environment.ProcessEnvironmentFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineDebugOptions;
/** Implementation of a {@link FlinkExecutableStageContext}. */
class FlinkDefaultExecutableStageContext implements
FlinkExecutableStageContext, AutoCloseable {
private final JobBundleFactory jobBundleFactory;
private static FlinkDefaultExecutableStageContext create(JobInfo jobInfo) {
-JobBundleFactory jobBundleFactory =
-DefaultJobBundleFactory.create(
-jobInfo,
-ImmutableMap.of(
-BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
-new DockerEnvironmentFactory.Provider(),
-BeamUrns.getUrn(StandardEnvironments.Environments.PROCESS),
-new ProcessEnvironmentFactory.Provider(),
-Environments.ENVIRONMENT_EMBEDDED, // Non Public urn for
testing.
-new EmbeddedEnvironmentFactory.Provider()));
-return new FlinkDefaultExecutableStageContext(jobBundleFactory);
+try {
+ PipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+ JobBundleFactory jobBundleFactory =
+ DefaultJobBundleFactory.create(
+ jobInfo,
+ ImmutableMap.of(
+ BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
+ new DockerEnvironmentFactory.Provider(
+ pipelineOptions
+ .as(PortablePipelineDebugOptions.class)
Review comment:
This line could be moved up to where the pipeline options are constructed.
It may also be nice to take care of the IOException try/catch clutter in
PipelineOptionsTranslation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151026)
Time Spent: 28h 20m (was: 28h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151023&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151023
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 04:09
Start Date: 04/Oct/18 04:09
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222532785
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions,
PipelineOptions {
+
+ @Description("Delete dynamically created SDK Harness containers.")
Review comment:
After reading the email thread, shouldn't this flag be
`retainDynamicSdkHarnessContainers` with default `false` - if so I agree it is
a debug option.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151023)
Time Spent: 28h 10m (was: 28h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h 10m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151008
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 04/Oct/18 02:14
Start Date: 04/Oct/18 02:14
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #6563:
[BEAM-4176] Cleanup SDK Harness docker container
URL: https://github.com/apache/beam/pull/6563#discussion_r222520317
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineDebugOptions.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Pipeline options for debug flags for portable runner. */
+@Experimental
+@Hidden
+public interface PortablePipelineDebugOptions extends ExperimentalOptions,
PipelineOptions {
+
+ @Description("Delete dynamically created SDK Harness containers.")
Review comment:
This doesn't seem to have anything to do with "debug". I think this should
live in PortablePipelineOptions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 151008)
Time Spent: 28h (was: 27h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 28h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151007&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151007 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 02:01 Start Date: 04/Oct/18 02:01 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-426860694 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151007) Time Spent: 27h 50m (was: 27h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 27h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151001&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151001 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 01:03 Start Date: 04/Oct/18 01:03 Worklog Time Spent: 10m Work Description: angoenka removed a comment on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-426851724 Python Flink ValidatesRunner Tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151001) Time Spent: 27h 40m (was: 27.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 27h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=151000&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-151000 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 01:03 Start Date: 04/Oct/18 01:03 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-426851807 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 151000) Time Spent: 27.5h (was: 27h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 27.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150999&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150999 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 01:02 Start Date: 04/Oct/18 01:02 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-426851724 Python Flink ValidatesRunner Tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150999) Time Spent: 27h 20m (was: 27h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 27h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150998&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150998 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 01:01 Start Date: 04/Oct/18 01:01 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563#issuecomment-426851550 R: @tweise @mxm @herohde This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150998) Time Spent: 27h 10m (was: 27h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Assignee: Ankur Goenka >Priority: Major > Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 > PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png > > Time Spent: 27h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150997&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150997 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Oct/18 00:53 Start Date: 04/Oct/18 00:53 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6563: [BEAM-4176] Cleanup SDK Harness docker container URL: https://github.com/apache/beam/pull/6563 We should clean the containers as they quickly pile up. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150997) Time Spent: 27h (was: 26h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug >
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150951&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150951
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 03/Oct/18 22:05
Start Date: 03/Oct/18 22:05
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6550: [BEAM-4176] Correctly
deserialize pipeline options on Fn harness
URL: https://github.com/apache/beam/pull/6550
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
index b47a40a55b7..6a8988e76fa 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
@@ -88,4 +88,16 @@ public static PipelineOptions fromProto(Struct protoOptions)
throws IOException
MAPPER.writeValueAsString(ImmutableMap.of("options", mapWithoutUrns)),
PipelineOptions.class);
}
+
+ /** Converts the provided Json{@link String} into {@link PipelineOptions}. */
+ public static PipelineOptions fromJson(String optionsJson) throws
IOException {
+Struct.Builder builder = Struct.newBuilder();
+JsonFormat.parser().merge(optionsJson, builder);
+return fromProto(builder.build());
+ }
+
+ /** Converts the provided {@link PipelineOptions} into Json{@link String}. */
+ public static String toJson(PipelineOptions options) throws IOException {
+return JsonFormat.printer().print(toProto(options));
+ }
}
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
index 4d832390fda..aef82d245bf 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
@@ -82,6 +82,17 @@ public void testToFromProto() throws Exception {
Struct reserializedStruct =
PipelineOptionsTranslation.toProto(deserializedStruct);
assertThat(reserializedStruct.getFieldsMap(),
equalTo(originalStruct.getFieldsMap()));
}
+
+@Test
+public void testToFromJson() throws Exception {
+ options.getOptionsId();
+ Struct originalStruct = PipelineOptionsTranslation.toProto(options);
+ String json = PipelineOptionsTranslation.toJson(options);
+
+ PipelineOptions deserializedOptions =
PipelineOptionsTranslation.fromJson(json);
+ Struct reserializedStruct =
PipelineOptionsTranslation.toProto(deserializedOptions);
+ assertThat(reserializedStruct.getFieldsMap(),
equalTo(originalStruct.getFieldsMap()));
+}
}
/** Tests that translations contain the correct contents. */
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
index 6ba55bca900..d1eb02d0538 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
@@ -18,7 +18,6 @@
package org.apache.beam.fn.harness;
-import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.collect.ImmutableList;
import java.util.EnumMap;
import java.util.List;
@@ -34,6 +33,7 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
import
org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder;
import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
@@ -43,7 +43,6 @@
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.vendor.protobuf.v3.com.google.protobuf.TextFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -88,11 +87,7 @@ public
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150882 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 03/Oct/18 18:29 Start Date: 03/Oct/18 18:29 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6550: [BEAM-4176] Correctly deserialize pipeline options on Fn harness URL: https://github.com/apache/beam/pull/6550#issuecomment-426746149 Yes, Launcher main method of FnHarness do not have good test coverage. I have moved this deserialization logic to PipelineOptionsTranslation and added the relevant test there. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150882) Time Spent: 26h 40m (was: 26.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 26h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150874
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 03/Oct/18 17:55
Start Date: 03/Oct/18 17:55
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222407020
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -116,6 +121,12 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() throws Exception {
+// Close the job service.
+try (CloseableResource jobService =
this.jobService) {}
Review comment:
I agree, I would differ this change to a later time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150874)
Time Spent: 26.5h (was: 26h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 26.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150873
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 03/Oct/18 17:53
Start Date: 03/Oct/18 17:53
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222406501
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -103,11 +112,7 @@ public State waitUntilFinish() {
response = stub.getState(request);
lastState = getJavaState(response.getState());
}
-try {
- jobService.close();
-} catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
-}
Review comment:
Makes sense
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150873)
Time Spent: 26h 20m (was: 26h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 26h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150783&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150783
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 03/Oct/18 15:29
Start Date: 03/Oct/18 15:29
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222357059
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -103,11 +112,7 @@ public State waitUntilFinish() {
response = stub.getState(request);
lastState = getJavaState(response.getState());
}
-try {
- jobService.close();
-} catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
-}
Review comment:
I think it would be ok to have a AutoCloseable close method and call it from
here to close eagerly when the terminal state has already been reached.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150783)
Time Spent: 26h 10m (was: 26h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 26h 10m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150782
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 03/Oct/18 15:26
Start Date: 03/Oct/18 15:26
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222356166
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -116,6 +121,12 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() throws Exception {
+// Close the job service.
+try (CloseableResource jobService =
this.jobService) {}
Review comment:
A background thread which does the polling might not be desired by the user.
On the other hand, when `waitUntilFinished()` is called polling is explicitly
requested. We would like to do the polling because it helps to find out whether
to close the JobServer connection but I think that needs to be handled in a
different way, e.g. with a registry of some sort.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150782)
Time Spent: 26h (was: 25h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 26h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150577 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 03/Oct/18 00:21 Start Date: 03/Oct/18 00:21 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6550: [BEAM-4176] Correctly deserialize pipeline options on Fn harness URL: https://github.com/apache/beam/pull/6550#issuecomment-426473092 R: @mxm @tweise @ryan-williams This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150577) Time Spent: 25h 50m (was: 25h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 25h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150576&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150576 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 03/Oct/18 00:20 Start Date: 03/Oct/18 00:20 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6550: [BEAM-4176] Correctly deserialize pipeline options on Fn harness URL: https://github.com/apache/beam/pull/6550 Desrialize the pipeline options using protobuf libraries Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150576) Time Spent: 25h 40m (was: 25.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150501
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 18:59
Start Date: 02/Oct/18 18:59
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222073006
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -116,6 +121,12 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() throws Exception {
+// Close the job service.
+try (CloseableResource jobService =
this.jobService) {}
Review comment:
Shall we go with this approach?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150501)
Time Spent: 25.5h (was: 25h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 25.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150500
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 18:58
Start Date: 02/Oct/18 18:58
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222072893
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -116,6 +121,12 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() throws Exception {
+// Close the job service.
+try (CloseableResource jobService =
this.jobService) {}
Review comment:
yes, we will need it. Also as the jobServer is not persistent, the
PipelineResult object usability is limited.
One approach can be to just keep on requesting for state after job creation.
Once we reach the termination state then close the connection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150500)
Time Spent: 25h 20m (was: 25h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 25h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150499
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 18:56
Start Date: 02/Oct/18 18:56
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r222072211
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -103,11 +112,7 @@ public State waitUntilFinish() {
response = stub.getState(request);
lastState = getJavaState(response.getState());
}
-try {
- jobService.close();
-} catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
-}
Review comment:
The current behavior of connection cleanup is un-intuitive. The connection
use to get closed only if the user called waitUntillFinish while if user did
not call WUF then the connection is not closed. Hence removing the connection
closing here so that the connection is only closed explicitly in close call.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150499)
Time Spent: 25h 10m (was: 25h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 25h 10m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150498&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150498
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 18:50
Start Date: 02/Oct/18 18:50
Worklog Time Spent: 10m
Work Description: angoenka commented on a change in pull request #6534:
[BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner
URL: https://github.com/apache/beam/pull/6534#discussion_r222069882
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java
##
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/** Category tag for tests which validate that currect failure message is
provided by failed
+ * pipeline. */
+public interface UsesFailureMessage extends NeedsRunner {}
Review comment:
Added the relevant usage of the annotation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150498)
Time Spent: 25h (was: 24h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 25h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150401&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150401
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 14:35
Start Date: 02/Oct/18 14:35
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6534:
[BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner
URL: https://github.com/apache/beam/pull/6534#discussion_r221976750
##
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java
##
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/** Category tag for tests which validate that currect failure message is
provided by failed
+ * pipeline. */
+public interface UsesFailureMessage extends NeedsRunner {}
Review comment:
This is not used anywhere. Was this intended?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150401)
Time Spent: 24h 50m (was: 24h 40m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 24h 50m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150386&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150386
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 14:01
Start Date: 02/Oct/18 14:01
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r221961766
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -116,6 +121,12 @@ public MetricResults metrics() {
throw new UnsupportedOperationException("Not yet implemented.");
}
+ @Override
+ public void close() throws Exception {
+// Close the job service.
+try (CloseableResource jobService =
this.jobService) {}
Review comment:
Wonder if we need some cleanup registry in the future to close all these
connections when we shutdown the JVM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150386)
Time Spent: 24.5h (was: 24h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 24.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150387
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 14:01
Start Date: 02/Oct/18 14:01
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r221954754
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -35,22 +35,28 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-class JobServicePipelineResult implements PipelineResult {
+class JobServicePipelineResult implements PipelineResult, AutoCloseable {
private static final long POLL_INTERVAL_MS = 10 * 1000;
private static final Logger LOG =
LoggerFactory.getLogger(JobServicePipelineResult.class);
private final ByteString jobId;
private final CloseableResource jobService;
+ @Nullable
+ private State terminationState;
JobServicePipelineResult(ByteString jobId,
CloseableResource jobService) {
this.jobId = jobId;
this.jobService = jobService;
+this.terminationState = null;
}
@Override
public State getState() {
+if(terminationState != null){
Review comment:
space missing after `if` and `)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150387)
Time Spent: 24.5h (was: 24h 20m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 24.5h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150385
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 14:01
Start Date: 02/Oct/18 14:01
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r221957558
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -103,11 +112,7 @@ public State waitUntilFinish() {
response = stub.getState(request);
lastState = getJavaState(response.getState());
}
-try {
- jobService.close();
-} catch (Exception e) {
- LOG.warn("Error cleaning up job service", e);
-}
Review comment:
Why should we keep the connection to the JobServer if we have reached the
final state? Final implies that no other state can be obtained.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150385)
Time Spent: 24h 20m (was: 24h 10m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 24h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150388
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 02/Oct/18 14:01
Start Date: 02/Oct/18 14:01
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #6537:
[BEAM-4176] Store and serve termination state after portable job termination
URL: https://github.com/apache/beam/pull/6537#discussion_r221954842
##
File path:
runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
##
@@ -89,6 +95,9 @@ public State waitUntilFinish(Duration duration) {
@Override
public State waitUntilFinish() {
+if(terminationState != null){
Review comment:
space missing after `if` and `)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 150388)
Time Spent: 24h 40m (was: 24.5h)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 24h 40m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150281 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 02/Oct/18 01:42 Start Date: 02/Oct/18 01:42 Worklog Time Spent: 10m Work Description: angoenka edited a comment on issue #6534: [BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner URL: https://github.com/apache/beam/pull/6534#issuecomment-426118658 R: @mxm @tweise @robertwb @ryan-williams This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150281) Time Spent: 24h 10m (was: 24h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 24h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150280 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 02/Oct/18 01:41 Start Date: 02/Oct/18 01:41 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6534: [BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner URL: https://github.com/apache/beam/pull/6534#issuecomment-426118658 R: @mxm @tweise @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150280) Time Spent: 24h (was: 23h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 24h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150276&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150276 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 02/Oct/18 01:08 Start Date: 02/Oct/18 01:08 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6537: [BEAM-4176] Store and serve termination state after portable job termination URL: https://github.com/apache/beam/pull/6537#issuecomment-426113332 R: @mxm @tweise This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150276) Time Spent: 23h 50m (was: 23h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 23h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150275&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150275 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 02/Oct/18 01:08 Start Date: 02/Oct/18 01:08 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6537: [BEAM-4176] Store and serve termination state after portable job termination URL: https://github.com/apache/beam/pull/6537 The jobserver connection is closed after job is terminated which means user can't check the job status again even when its terminated. Store and provide terminated job status to user. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150275) Time Spent: 23h 40m (was: 23.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > >
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=150257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-150257 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 01/Oct/18 23:23 Start Date: 01/Oct/18 23:23 Worklog Time Spent: 10m Work Description: angoenka opened a new pull request #6534: [BEAM-4176] Filtering Metrics and Splittable ParDo tests for Portable Runner URL: https://github.com/apache/beam/pull/6534 Metrics and Splittable ParDo are not yet supported on portable runners. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 150257) Time Spent: 23.5h (was: 23h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Pr
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=146064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146064 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 20/Sep/18 17:05 Start Date: 20/Sep/18 17:05 Worklog Time Spent: 10m Work Description: tweise commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-423259567 @ryan-williams how to run the PVR tests: `Run Python Flink PortableValidatesRunner` see https://github.com/apache/beam/pull/6340 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146064) Time Spent: 23h 20m (was: 23h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 23h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=146063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146063
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 20/Sep/18 16:57
Start Date: 20/Sep/18 16:57
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6225: [BEAM-4176] move some
artifact-staging logs from info to debug
URL: https://github.com/apache/beam/pull/6225
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
index cc7b2feadc7..d5dbca0c23f 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
@@ -28,6 +28,7 @@
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.security.MessageDigest;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
@@ -57,12 +58,16 @@
import org.apache.beam.vendor.grpc.v1.io.grpc.Channel;
import org.apache.beam.vendor.grpc.v1.io.grpc.stub.StreamObserver;
import org.apache.beam.vendor.protobuf.v3.com.google.protobuf.ByteString;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** A client to stage files on an {@link ArtifactStagingServiceGrpc
ArtifactService}. */
public class ArtifactServiceStager {
// 2 MB per file-request
private static final int DEFAULT_BUFFER_SIZE = 2 * 1024 * 1024;
+ private static final Logger LOG =
LoggerFactory.getLogger(ArtifactServiceStager.class);
+
public static ArtifactServiceStager overChannel(Channel channel) {
return overChannel(channel, DEFAULT_BUFFER_SIZE);
}
@@ -94,9 +99,10 @@ private ArtifactServiceStager(Channel channel, int
bufferSize) {
*
* @return The artifact staging token returned by the service
*/
- public String stage(String stagingSessionToken, Iterable files)
+ public String stage(String stagingSessionToken, Collection files)
throws IOException, InterruptedException {
final Map> futures = new
HashMap<>();
+LOG.info("Staging {} files (token: {})", files.size(),
stagingSessionToken);
for (StagedFile file : files) {
futures.put(
file,
@@ -114,6 +120,8 @@ private String stageManifest(
try {
StagingResult stagingResult = MoreFutures.get(stagingFuture);
if (stagingResult.isSuccess()) {
+LOG.info(
+"Staged {} files (token: {})", stagingResult.getMetadata().size(),
stagingSessionToken);
Manifest manifest =
Manifest.newBuilder().addAllArtifact(stagingResult.getMetadata()).build();
CommitManifestResponse response =
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
index 613ec4d4dd7..b4482d7e4fc 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
@@ -100,7 +100,7 @@ public void getManifest(
public void getArtifact(
ArtifactApi.GetArtifactRequest request,
StreamObserver responseObserver) {
-LOG.info("GetArtifact {}", request);
+LOG.debug("GetArtifact {}", request);
String name = request.getName();
try {
ArtifactApi.ProxyManifest proxyManifest =
MANIFEST_CACHE.get(request.getRetrievalToken());
@@ -131,7 +131,7 @@ public void getArtifact(
ResourceId artifactResourceId =
FileSystems.matchNewResource(location.getUri(), false /* is
directory */);
- LOG.info("Artifact {} located in {}", name, artifactResourceId);
+ LOG.debug("Artifact {} located in {}", name, artifactResourceId);
Hasher hasher = Hashing.md5().newHasher();
byte[] data = new byte[ARTIFACT_CHUNK_SIZE_BYTES];
try (InputStream stream =
Channels.newInputStream(FileSystems.open(artifactResourceId))) {
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
b/runn
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=146026&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146026 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 20/Sep/18 14:13 Start Date: 20/Sep/18 14:13 Worklog Time Spent: 10m Work Description: mxm commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-423198886 I think Thomas is alluding to https://issues.apache.org/jira/browse/BEAM-5441. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146026) Time Spent: 23h (was: 22h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 23h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=146023&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146023 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 20/Sep/18 13:59 Start Date: 20/Sep/18 13:59 Worklog Time Spent: 10m Work Description: tweise commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-423193947 For this category of changes, we should in the future run the post commit PVR tests in Jenkins before merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 146023) Time Spent: 22h 50m (was: 22h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 22h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145695 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 19/Sep/18 15:20 Start Date: 19/Sep/18 15:20 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6225: [BEAM-4176] move some artifact-staging logs from info to debug URL: https://github.com/apache/beam/pull/6225#issuecomment-422844406 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145695) Time Spent: 22h 40m (was: 22.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 22h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
***UNCHECKED*** [jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145690 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 19/Sep/18 15:03 Start Date: 19/Sep/18 15:03 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6225: [BEAM-4176] move some artifact-staging logs from info to debug URL: https://github.com/apache/beam/pull/6225#issuecomment-422838244 several tests failed that I don't think are related to my changes here: `:beam-sdks-java-io-hadoop-input-format:test`: ``` 19:51:52 localhost/127.0.0.1:7000 is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services 19:51:52 Fatal configuration error; unable to start server. See log for stacktrace. 19:51:52 Sep 18, 2018 11:51:51 PM org.apache.cassandra.service.CassandraDaemon exitOrFail 19:51:52 SEVERE: Fatal configuration error 19:51:52 org.apache.cassandra.exceptions.ConfigurationException: localhost/127.0.0.1:7000 is in use by another process. Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services 19:51:52 at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:557) 19:51:52 at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:501) 19:51:52 at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:485) 19:51:52 at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:745) ``` `:beam-sdks-java-io-elasticsearch-tests-2:test`: ``` 19:45:44 org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest > testWriteWithIndexFn FAILED 19:45:44 java.lang.AssertionError: Einstein index holds incorrect count expected:<10> but was:<20> 19:45:44 at org.junit.Assert.fail(Assert.java:88) 19:45:44 at org.junit.Assert.failNotEquals(Assert.java:834) 19:45:44 at org.junit.Assert.assertEquals(Assert.java:645) 19:45:44 at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.testWriteWithIndexFn(ElasticsearchIOTestCommon.java:439) 19:45:44 at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIndexFn(ElasticsearchIOTest.java:175) ``` `:beam-sdks-java-io-amazon-web-services:test`: ``` 19:43:59 org.apache.beam.sdk.io.aws.sqs.SqsIOTest > testRead FAILED 19:43:59 akka.stream.impl.io.ConnectionSourceStage$$anon$2$$anon$1: Bind failed because of Address already in use 19:43:59 19:43:59 Caused by: 19:43:59 java.net.BindException: Address already in use 19:44:07 19:44:07 org.apache.beam.sdk.io.aws.sns.SnsIOTest > testRetries STANDARD_ERROR 19:44:07 Sep 18, 2018 11:44:07 PM org.apache.beam.sdk.io.aws.sns.SnsIO$Write$SnsWriterFn processElement 19:44:07 WARNING: Error writing to SNS. Retry attempt[1] 19:44:07 com.amazonaws.services.sns.model.InternalErrorException: Service unavailable (Service: null; Status Code: 0; Error Code: null; Request ID: null) 19:44:07 at org.apache.beam.sdk.io.aws.sns.AmazonSNSMockErrors.publish(AmazonSNSMockErrors.java:29) 19:44:07 at org.apache.beam.sdk.io.aws.sns.SnsIO$Write$SnsWriterFn.processElement(SnsIO.java:314) ``` perhaps the hadoop and AWS modules were conflicting about binding the same port? I'm not sure what happened to the elasticsearch test This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145690) Time Spent: 22.5h (was: 22h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 22.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145521&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145521 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 23:10 Start Date: 18/Sep/18 23:10 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422588775 Yea, to be clear, the tests themselves (apparently) weren't getting stuck, but attempting to run them was, due to local Docker issues, which I agree is preferable š This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145521) Time Spent: 22h 20m (was: 22h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 22h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145510&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145510 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 22:36 Start Date: 18/Sep/18 22:36 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422581022 Thanks Ryan! Also the good part is that the test cases are not getting stuck. Many a times the test use to get stuck instead of failing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145510) Time Spent: 22h 10m (was: 22h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 22h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145506
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 18/Sep/18 22:15
Start Date: 18/Sep/18 22:15
Worklog Time Spent: 10m
Work Description: lukecwik closed pull request #6328: [BEAM-4176]
enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/NativeTransforms.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/NativeTransforms.java
new file mode 100644
index 000..730a5ca561f
--- /dev/null
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/NativeTransforms.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import java.util.Iterator;
+import java.util.ServiceLoader;
+import java.util.function.Predicate;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+
+/**
+ * An extension point for users to define their own native transforms for
usage with specific
+ * runners. This extension point enables shared libraries within the Apache
Beam codebase to treat
+ * the native transform as a primitive transforms that the runner implicitly
understands.
+ *
+ * Warning:Usage of native transforms within pipelines will prevent
users from migrating
+ * between runners as there is no expectation that the transform will be
understood by all runners.
+ * Note that for some use cases this can be a way to test out a new type of
transform on a limited
+ * set of runners and promote its adoption as a primitive within the Apache
Beam model.
+ *
+ * Note that users are required to ensure that translation and execution
for the native transform
+ * is supported by their runner.
+ *
+ * Automatic registration occurs by creating a {@link ServiceLoader} entry
and a concrete
+ * implementation of the {@link IsNativeTransform} interface. It is optional
but recommended to use
+ * one of the many build time tools such as {@link AutoService} to generate
the necessary META-INF
+ * files automatically.
+ */
+public class NativeTransforms {
+ /**
+ * Returns true if an only if the Runner understands this transform and can
handle it directly.
+ */
+ public static boolean isNative(RunnerApi.PTransform pTransform) {
+Iterator matchers =
ServiceLoader.load(IsNativeTransform.class).iterator();
+while (matchers.hasNext()) {
+ if (matchers.next().test(pTransform)) {
+return true;
+ }
+}
+return false;
+ }
+
+ /** A predicate which returns true if and only if the transform is a native
transform. */
+ public interface IsNativeTransform extends Predicate {
+@Override
+boolean test(RunnerApi.PTransform pTransform);
+ }
+}
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 7be45388695..0401b572dda 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -67,7 +67,14 @@
getUrn(StandardPTransforms.Primitives.ASSIGN_WINDOWS);
public static final String TEST_STREAM_TRANSFORM_URN =
getUrn(StandardPTransforms.Primitives.TEST_STREAM);
+ public static final String MAP_WINDOWS_TRANSFORM_URN =
+ getUrn(StandardPTransforms
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145505&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145505 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 22:15 Start Date: 18/Sep/18 22:15 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422575816 Thanks Ryan. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145505) Time Spent: 21h 50m (was: 21h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145504&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145504 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 22:09 Start Date: 18/Sep/18 22:09 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422573991 OK, I managed to get [a run to complete](https://storage.googleapis.com/runsascoded-tmp/beam-portable-flink-vr-tests/ecd1ac085a/index.html):  Notes on getting it to run: - I increased my local Docker image size to 256GB (from 64GB) - I also ran `docker container prune -f` multiple times during the run - idk if either of the above are necessary or sufficient - `docker system df` is also useful to determine how full the image is. Notes on the results: - 10 more cases are passing since I started this PR - possibly due to fixes I added for unittests over the course of this PR - or other OOB changes - runtime is much worse! 2h30m. seems like nothing to be done there atm, and is related to an increased number of containers being created (thanks for explaining @angoenka) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145504) Time Spent: 21h 40m (was: 21.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145473 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 20:59 Start Date: 18/Sep/18 20:59 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422552764 It seems like we should be able to reuse a single docker container image for all the test runs instead of having each test create one. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145473) Time Spent: 21.5h (was: 21h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145409 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 18:04 Start Date: 18/Sep/18 18:04 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422491341 its in this commit https://github.com/apache/beam/commit/992ece0a4ad444dee9faea21e4729c495dea6b42 I don't think this will make a lot of difference though as we anyways create 200+ containers, 1 for each test atlease. The best way to go about it is to use process based SDK Harness which is in development at the moment https://github.com/apache/beam/pull/6373 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145409) Time Spent: 21h 20m (was: 21h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145408 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:59 Start Date: 18/Sep/18 17:59 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422489947 ah, yea `docker container ls -a` seems to show old containers for every test-case, so that sounds like a good theory about the problem is there something better to do than manually clean those a few times while PVR tests are running? do you have a link handy to where we disabled the cleanup? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145408) Time Spent: 21h 10m (was: 21h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145404&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145404 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:56 Start Date: 18/Sep/18 17:56 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422488926 We stopped doing lazy cleanup of docker containers which might have a minor impact of the number of containers created and hence their memory footprint. With 200+ test cases we are expecting atleast 200 containers anyways. We are introducing process based sdkhanress which should remove creation of docker containers whole together. For now please try cleaning the dockers containers and see if it helps. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145404) Time Spent: 21h (was: 20h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 21h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145399 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:43 Start Date: 18/Sep/18 17:43 Worklog Time Spent: 10m Work Description: ryan-williams edited a comment on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422480207 My working theory is: - something has caused us to use dramatically more of Docker's "virtual" space (not sure this is the right term; it's the "disk image size" in preferences, also discussed [here](https://forums.docker.com/t/no-space-left-on-device-error/10894)) - PVR tests still pass or fail in isolation as one would expect (and presumably we're still down to 127 failures, as we were at the start of this PR) - however, once you've run a certain total number of tests, your local Docker space will be full and everything will fail from that point on Debugging this doesn't _have_ to hold up this PR, but I don't think there's a huge rush to get it in either, so I can spend some more time on it. I've just resized my local Docker image disk space to 256GB (from 64GB) and am running again to see if it gets further This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145399) Time Spent: 20h 50m (was: 20h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 20h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145397 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:43 Start Date: 18/Sep/18 17:43 Worklog Time Spent: 10m Work Description: ryan-williams edited a comment on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422480207 My working theory is: - something has caused us to use dramatically more of Docker's "virtual" space (not sure this is the right term) - PVR tests still pass or fail in isolation as one would expect (and presumably we're still down to 127 failures, as we were at the start of this PR) - however, once you've run a certain total number of tests, your local Docker space will be full and everything will fail from that point on Debugging this doesn't _have_ to hold up this PR, but I don't think there's a huge rush to get it in either, so I can spend some more time on it. I've just resized my local Docker image disk space to 256GB (from 64GB) and am running again to see if it gets further This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145397) Time Spent: 20h 40m (was: 20.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 20h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145389&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145389 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:30 Start Date: 18/Sep/18 17:30 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422480207 My working theory is: - something has caused us to use dramatically more of Docker's virtual space - PVR tests still pass or fail in isolation as one would expect (and presumably we're still down to 127 failures, as we were at the start of this PR) - however, once you've run a certain total number of tests, your local Docker space will be full and everything will fail from that point on Debugging this doesn't _have_ to hold up this PR, but I don't think there's a huge rush to get it in either, so I can spend some more time on it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145389) Time Spent: 20.5h (was: 20h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 20.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145385 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:24 Start Date: 18/Sep/18 17:24 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6225: [BEAM-4176] move some artifact-staging logs from info to debug URL: https://github.com/apache/beam/pull/6225#issuecomment-422478212 I suggest you follow the same pattern like Dataflow has been doing: https://github.com/apache/beam/blob/fef40fc60cbd372a7ea89dc8142694d16476c6ce/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L315 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145385) Time Spent: 20h 20m (was: 20h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 20h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145383 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 17:23 Start Date: 18/Sep/18 17:23 Worklog Time Spent: 10m Work Description: angoenka commented on issue #6225: [BEAM-4176] move some artifact-staging logs from info to debug URL: https://github.com/apache/beam/pull/6225#issuecomment-422477824 Yes, they seem to be a bit spammy. Seems reasonable. Lets move them to debug. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145383) Time Spent: 20h 10m (was: 20h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 20h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145380&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145380
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 18/Sep/18 17:17
Start Date: 18/Sep/18 17:17
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6328:
[BEAM-4176] enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328#discussion_r218520440
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/NativeTransformsTest.java
##
@@ -39,14 +40,15 @@ public boolean test(RunnerApi.PTransform pTransform) {
@Test
public void testMatch() {
-NativeTransforms.isNative(
-RunnerApi.PTransform.newBuilder()
-
.setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn("test").build())
-.build());
+Assert.assertTrue(
Review comment:
What a silly mistake I made
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 145380)
Time Spent: 20h (was: 19h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 20h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145359&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145359 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 16:40 Start Date: 18/Sep/18 16:40 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422464153 Seeing significant perf degradations running the Java PVR tests now, presumably due to unrelated changes that have gone in since I opened this PR. I can't get through a run of all the tests without a hang; [in some (all?) cases it seems that Docker is running out of space in its virtual filesystem](https://gist.github.com/ryan-williams/45f3b8e4f67aca28acbc2f1ee733fd47#file-gistfile1-txt-L17). Previously all 200+ cases ran in a bit over an hour, but now it grinds to a halt about 50 cases in (always on a different test case, n=3) in about an hour. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145359) Time Spent: 19h 50m (was: 19h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 19h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145163&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145163 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 00:40 Start Date: 18/Sep/18 00:40 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6225: [BEAM-4176] move some artifact-staging logs from info to debug URL: https://github.com/apache/beam/pull/6225#issuecomment-422215682 Finally coming back to this; [I tried adding an SDK-side log as you suggested](https://github.com/ryan-williams/beam/commit/863f3706bac051adc07778ba7cc25c48d4795163#diff-968a6a22041ca278b1b1ed91603639d8R193), but it feels equivalently spammy to me at first glance: [](https://cl.ly/3bfb319aa212/Screen%20Shot%202018-09-17%20at%208.37.56%20PM.png) do you think it's important to INFO-log every artifact? I'd have said these should be DEBUGs on both ends perhaps two INFOs saying "staging X artifacts" / "staged X artifacts"? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145163) Time Spent: 19h 40m (was: 19.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 19h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145162 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 18/Sep/18 00:37 Start Date: 18/Sep/18 00:37 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422215195 I didn't have docker daemon running :) a re-run hung in `FlattenTest` though, but I'm assuming that's not related to this changeā¦ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145162) Time Spent: 19.5h (was: 19h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 19.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145079 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 17/Sep/18 21:36 Start Date: 17/Sep/18 21:36 Worklog Time Spent: 10m Work Description: ryan-williams commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-422180055 I just ran VR tests again on this PR and am seeing the original failure count; I suspect I made a mistake running the tests, but will update here as I learn more. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 145079) Time Spent: 19h 20m (was: 19h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 19h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=144454&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-144454 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 14/Sep/18 20:28 Start Date: 14/Sep/18 20:28 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-421475492 Based upon community feedback, I opened up https://github.com/ryan-williams/beam/pull/2 against your branch. Please merge or advise on different course of action. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 144454) Time Spent: 19h 10m (was: 19h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 19h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=142534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-142534
]
ASF GitHub Bot logged work on BEAM-4176:
Author: ASF GitHub Bot
Created on: 09/Sep/18 19:51
Start Date: 09/Sep/18 19:51
Worklog Time Spent: 10m
Work Description: ryan-williams commented on issue #6328: [BEAM-4176]
enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328#issuecomment-419740410
Looking into [some remaining failures in
ReferenceRunnerTest](https://storage.googleapis.com/runsascoded-tmp/beam/ReferenceRunnerTest-1536522471/classes/org.apache.beam.runners.direct.portable.ReferenceRunnerTest.html#pipelineExecution):
**pipelineExecution** / **GBK**:
```
java.lang.NullPointerException: No evaluator for PTransform
"beam:transform:group_by_key:v1"
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:864)
at
org.apache.beam.runners.direct.portable.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:96)
at
org.apache.beam.runners.direct.portable.DirectTransformExecutor.run(DirectTransformExecutor.java:93)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
```
**testSDF**:
```
java.lang.IllegalArgumentException: PCollectionNodes
[PCollectionNode{id=ParDo(PairStringWithIndexToLength)/ParMultiDo(PairStringWithIndexToLength).output,
PCollection=unique_name:
"ParDo(PairStringWithIndexToLength)/ParMultiDo(PairStringWithIndexToLength).output"
coder_id: "KvCoder7"
is_bounded: UNBOUNDED
windowing_strategy_id: "WindowingStrategy(GlobalWindows)"
}] were consumed but never produced
at
org.apache.beam.repackaged.beam_runners_core_construction_java.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.buildNetwork(QueryablePipeline.java:183)
at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.(QueryablePipeline.java:106)
at
org.apache.beam.runners.core.construction.graph.QueryablePipeline.forPrimitivesIn(QueryablePipeline.java:69)
at
org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.(GreedyPipelineFuser.java:67)
at
org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:89)
at
org.apache.beam.runners.direct.portable.ReferenceRunner.executable(ReferenceRunner.java:128)
at
org.apache.beam.runners.direct.portable.ReferenceRunner.(ReferenceRunner.java:102)
at
org.apache.beam.runners.direct.portable.ReferenceRunner.forInProcessPipeline(ReferenceRunner.java:114)
at
org.apache.beam.runners.direct.portable.ReferenceRunnerTest.testSDF(ReferenceRunnerTest.java:207)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 142534)
Time Spent: 19h (was: 18h 50m)
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
>
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 19h
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=142228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-142228 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 07/Sep/18 15:51 Start Date: 07/Sep/18 15:51 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-419483192 Will bring to ML This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 142228) Time Spent: 18h 50m (was: 18h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=142191&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-142191 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 07/Sep/18 14:13 Start Date: 07/Sep/18 14:13 Worklog Time Spent: 10m Work Description: tweise commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-419452647 @lukecwik this is probably something we should take to the mailing list for broader visibility. Currently the Flink runner does the pipeline surgery before translation. With the proposed change the customization referenced above would need to spread over multiple components (fuser/prep and translator) while it's encapsulated in a single place now. In the example the Flink sources really are primitives that should remain handled on the runner side as such (effectively like ParDo without inputs). Would it perhaps also make sense to assume that as a default when our common fusion code does not recognize a URN? If I was to introduce a custom URN that requires fusion break, then yes, it would be necessary to provide that information to the fuser. That seems cleaner than the runner temporarily switching URNs to appease the fuser. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 142191) Time Spent: 18h 40m (was: 18.5h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18h 40m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=141065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141065 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Sep/18 22:03 Start Date: 04/Sep/18 22:03 Worklog Time Spent: 10m Work Description: lukecwik edited a comment on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-418532499 The shared libraries need to understand all the primitives given to them as they are required to satisfy the contracts that each primitive has. The code within QueryablePipeline/ExecutableStage/GreedyPipelineFuser/... make assumptions around what neighboring primitives exist and how they are compatible with each other. The pipeline representation can use pardo/read/... primitives which the runner intercepts during translation *OR* the runner can choose to intercept the pipeline before it gives it to the shared libraries to do fusion/... and update the representation to be compatible with those libraries *OR* those libraries need to expose hooks for runner known primitives to describe how they should treat these primitives (e.g. should they break fusion, or be fusable with the SDK transforms). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141065) Time Spent: 18.5h (was: 18h 20m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18.5h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=141063&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141063 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Sep/18 22:01 Start Date: 04/Sep/18 22:01 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-418532499 The shared libraries need to understand all the primitives given to them as they are required to satisfy the contracts that each primitive has. The code within QueryablePipeline/ExecutableStage/GreedyPipelineFuser/... make assumptions around what neighboring primitives exist and how they are compatible with each other. The runner can choose to intercept the pipeline before it gives it to the shared libraries to do fusion/... and update the representation to be compatible with those libraries or those libraries need to expose hooks for runner known primitives to describe how they should treat these primitives (e.g. should they break fusion, or be fusable with the SDK transforms). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141063) Time Spent: 18h 20m (was: 18h 10m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18h 20m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=141059&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141059 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Sep/18 21:48 Start Date: 04/Sep/18 21:48 Worklog Time Spent: 10m Work Description: tweise commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-418529203 Can you elaborate what the need for this change is? (It's not obvious why the use of custom URNs should be restricted.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141059) Time Spent: 18h 10m (was: 18h) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18h 10m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=141056&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141056 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Sep/18 21:45 Start Date: 04/Sep/18 21:45 Worklog Time Spent: 10m Work Description: tweise commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-418528410 @lukecwik here is the example: https://github.com/lyft/beam/blob/release-2.8.0-lyft/sdks/python/custom-source-example.py and the translator extension: https://github.com/lyft/beam/blob/release-2.8.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java#L52 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141056) Time Spent: 18h (was: 17h 50m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 18h > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
[ https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=141055&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141055 ] ASF GitHub Bot logged work on BEAM-4176: Author: ASF GitHub Bot Created on: 04/Sep/18 21:34 Start Date: 04/Sep/18 21:34 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #6328: [BEAM-4176] enumerate primitive transforms in portable construction URL: https://github.com/apache/beam/pull/6328#issuecomment-418525720 Ryan, as a follow-up to this change it would be good to make unknown transforms that aren't composites an error. This would be any transform that returns a different PCollection then one it consumed if the URN is not a primitive. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 141055) Time Spent: 17h 50m (was: 17h 40m) > Java: Portable batch runner passes all ValidatesRunner tests that > non-portable runner passes > > > Key: BEAM-4176 > URL: https://issues.apache.org/jira/browse/BEAM-4176 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Ben Sidhom >Priority: Major > Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot > 2018-09-03 at 11.07.38 AM.png > > Time Spent: 17h 50m > Remaining Estimate: 0h > > We need this as a sanity check that runner execution is correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
