[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361456&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361456 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 19/Dec/19 13:34 Start Date: 19/Dec/19 13:34 Worklog Time Spent: 10m Work Description: lgajowy commented on pull request #10423: Revert "[BEAM-5495] Use ClassGraph to detect classpath resources" URL: https://github.com/apache/beam/pull/10423 This reverts commit 7e5a885d854eb86b7dd07f778c05fa5d595adc7d. Classgraph seems to suffer some problems tha appeared in post commit testing (see: https://github.com/apache/beam/pull/10410). We can switch back to one of previously proposed solutions without reverting the whole branch but only one commit that introduces classgraph Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunne
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361403 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 18/Dec/19 23:03 Start Date: 18/Dec/19 23:03 Worklog Time Spent: 10m Work Description: TheNeuralBit commented on pull request #10420: Revert "Merge pull request #10268: [BEAM-5495] Adapt PipelineResource… URL: https://github.com/apache/beam/pull/10420 …s to be compatible with Java 11" This reverts commit e4a3594c69210706c00a0e13f11c2e21000a3cea, reversing changes made to 15a3eb06e5fbfebf1592a3440fd2c84359e4a4be. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[ PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361365 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 18/Dec/19 13:25 Start Date: 18/Dec/19 13:25 Worklog Time Spent: 10m Work Description: lgajowy commented on pull request #10410: [BEAM-5495] PR 10268 followup URL: https://github.com/apache/beam/pull/10410 Since PR-10268 was reviewed, merged and then reviewed again, I'm posting a follow-up PR to address the comments from the last review. I still **have not** adressed 2 comments: - https://github.com/apache/beam/pull/10268#discussion_r359024833 (still investigating) - https://github.com/apache/beam/pull/10268#discussion_r359023901 (waiting for reviewer response) We can either fix them in this or some other prs later (if needed). R: @lukecwik R: @mxm Thanks! Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[ PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361108&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361108
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359019634
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesOptions.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import io.github.classgraph.ClassGraph;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.InstanceBuilder;
+
+/** Pipeline options dedicated to detecting classpath resources. */
+public interface PipelineResourcesOptions extends PipelineOptions {
+
+ @Description(
+ "The class of the pipeline resources detector factory that should be
created and used to create "
+ + "the detector. If not set explicitly, a default class will be used
to instantiate the factory.")
+ @Default.Class(ClasspathScanningResourcesDetectorFactory.class)
+ Class
+ getPipelineResourcesDetectorFactoryClass();
+
+ void setPipelineResourcesDetectorFactoryClass(
+ Class factoryClass);
+
+ @JsonIgnore
+ @Description(
+ "Instance of a pipeline resources detection algorithm. If not set
explicitly, a default implementation will be used")
+ @Default.InstanceFactory(PipelineResourcesDetectorFactory.class)
Review comment:
its a minor hassle but please duplicate the description as a javadoc comment
on the getter so that the javadoc has this information as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361108)
Time Spent: 15h (was: 14h 50m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361114
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359026175
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesDetectorAbstractFactory.java
##
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+/** Provides pipeline resources detection algorithm. */
+public interface PipelineResourcesDetectorAbstractFactory {
Review comment:
This isn't an abstract class but an interface, please rename to
`PiplineResourcesDetectorFactory` and/or make it an inner interface of
`PipelineResourcesDetector` and just call it `Factory` there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361114)
Time Spent: 15h 40m (was: 15.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=36&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-36
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359023901
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##
@@ -28,51 +32,44 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
/** Tests for PipelineResources. */
@RunWith(JUnit4.class)
public class PipelineResourcesTest {
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
- @Rule public transient ExpectedException thrown = ExpectedException.none();
@Test
- public void detectClassPathResourceWithFileResources() throws Exception {
+ public void testDetectsResourcesToStage() throws IOException {
Review comment:
Please re-add that the resources are detected in the order they appear as
part of the URLClassLoader.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 36)
Time Spent: 15h 20m (was: 15h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361106
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359018486
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesOptions.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import io.github.classgraph.ClassGraph;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.InstanceBuilder;
+
+/** Pipeline options dedicated to detecting classpath resources. */
+public interface PipelineResourcesOptions extends PipelineOptions {
+
+ @Description(
+ "The class of the pipeline resources detector factory that should be
created and used to create "
+ + "the detector. If not set explicitly, a default class will be used
to instantiate the factory.")
+ @Default.Class(ClasspathScanningResourcesDetectorFactory.class)
+ Class
+ getPipelineResourcesDetectorFactoryClass();
+
+ void setPipelineResourcesDetectorFactoryClass(
+ Class factoryClass);
+
+ @JsonIgnore
+ @Description(
+ "Instance of a pipeline resources detection algorithm. If not set
explicitly, a default implementation will be used")
+ @Default.InstanceFactory(PipelineResourcesDetectorFactory.class)
+ PipelineResourcesDetector getPipelineResourcesDetector();
+
+ void setPipelineResourcesDetector(PipelineResourcesDetector
pipelineResourcesDetector);
+
+ class PipelineResourcesDetectorFactory implements
DefaultValueFactory {
Review comment:
class comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361106)
Time Spent: 14h 50m (was: 14h 40m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 14h 50m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361110
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359019828
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesOptions.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import io.github.classgraph.ClassGraph;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.InstanceBuilder;
+
+/** Pipeline options dedicated to detecting classpath resources. */
+public interface PipelineResourcesOptions extends PipelineOptions {
+
+ @Description(
+ "The class of the pipeline resources detector factory that should be
created and used to create "
+ + "the detector. If not set explicitly, a default class will be used
to instantiate the factory.")
+ @Default.Class(ClasspathScanningResourcesDetectorFactory.class)
Review comment:
Add `@JsonIgnore`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361110)
Time Spent: 15h 10m (was: 15h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h 10m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361109&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361109
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359021220
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/ClasspathScanningResourcesDetectorTest.java
##
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.not;
+import static org.junit.Assert.assertFalse;
+
+import io.github.classgraph.ClassGraph;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.List;
+import java.util.jar.JarOutputStream;
+import java.util.jar.Manifest;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.testing.RestoreSystemProperties;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.mockito.Mockito;
+
+public class ClasspathScanningResourcesDetectorTest {
+
+ @Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
+
+ @Rule public transient RestoreSystemProperties systemProperties = new
RestoreSystemProperties();
+
+ private ClasspathScanningResourcesDetector detector;
+
+ private ClassLoader classLoader;
Review comment:
classLoader and detector are assigned but are not shared outside of the
method body so please create them within the test instead as local variables.
This would help with test readability.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361109)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361105
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359017865
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesDetector.java
##
@@ -18,10 +18,10 @@
package org.apache.beam.runners.core.construction.resources;
import java.io.Serializable;
-import java.util.List;
+import java.util.stream.Stream;
/** Interface for an algorithm detecting classpath resources for pipelines. */
public interface PipelineResourcesDetector extends Serializable {
- List detect(ClassLoader classLoader);
+ Stream detect(ClassLoader classLoader);
Review comment:
I would have also preferred list since stream can imply that it is
infinitely long since there could be a generator function the person
implements. Lists are ordered and have finite size.
Also, set is the wrong abstraction since we want to maintain the classpath
order and we also want to maintain duplicates.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361105)
Time Spent: 14h 40m (was: 14.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 14h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361112
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359019575
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesOptions.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import io.github.classgraph.ClassGraph;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.InstanceBuilder;
+
+/** Pipeline options dedicated to detecting classpath resources. */
+public interface PipelineResourcesOptions extends PipelineOptions {
+
+ @Description(
+ "The class of the pipeline resources detector factory that should be
created and used to create "
Review comment:
its a minor hassle but please duplicate the description as a javadoc comment
on the getter so that the javadoc has this information as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361112)
Time Spent: 15.5h (was: 15h 20m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15.5h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361113&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361113
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359024833
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##
@@ -49,7 +50,19 @@
ClassLoader classLoader, PipelineOptions options) {
PipelineResourcesOptions artifactsRelatedOptions =
options.as(PipelineResourcesOptions.class);
-return
artifactsRelatedOptions.getPipelineResourcesDetector().detect(classLoader);
+return artifactsRelatedOptions
+.getPipelineResourcesDetector()
+.detect(classLoader)
+.filter(isStageable())
+.collect(Collectors.toList());
+ }
+
+ /**
+ * Returns a predicate for filtering all resources that are impossible to
stage (like gradle
+ * wrapper jars).
+ */
+ private static Predicate isStageable() {
+return resourcePath -> !resourcePath.contains("gradle/wrapper");
Review comment:
Instead of blacklisting, why does gradle appear on the classpath in the
first place?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361113)
Time Spent: 15h 40m (was: 15.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 15h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=361107&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361107
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:02
Start Date: 17/Dec/19 21:02
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r359018417
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesOptions.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.resources;
+
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import io.github.classgraph.ClassGraph;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.InstanceBuilder;
+
+/** Pipeline options dedicated to detecting classpath resources. */
+public interface PipelineResourcesOptions extends PipelineOptions {
+
+ @Description(
+ "The class of the pipeline resources detector factory that should be
created and used to create "
+ + "the detector. If not set explicitly, a default class will be used
to instantiate the factory.")
+ @Default.Class(ClasspathScanningResourcesDetectorFactory.class)
+ Class
+ getPipelineResourcesDetectorFactoryClass();
+
+ void setPipelineResourcesDetectorFactoryClass(
+ Class factoryClass);
+
+ @JsonIgnore
+ @Description(
+ "Instance of a pipeline resources detection algorithm. If not set
explicitly, a default implementation will be used")
+ @Default.InstanceFactory(PipelineResourcesDetectorFactory.class)
+ PipelineResourcesDetector getPipelineResourcesDetector();
+
+ void setPipelineResourcesDetector(PipelineResourcesDetector
pipelineResourcesDetector);
+
+ class PipelineResourcesDetectorFactory implements
DefaultValueFactory {
+
+@Override
+public PipelineResourcesDetector create(PipelineOptions options) {
+ PipelineResourcesOptions resourcesOptions =
options.as(PipelineResourcesOptions.class);
+
+ PipelineResourcesDetectorAbstractFactory resourcesToStage =
+
InstanceBuilder.ofType(PipelineResourcesDetectorAbstractFactory.class)
+
.fromClass(resourcesOptions.getPipelineResourcesDetectorFactoryClass())
+ .fromFactoryMethod("create")
+ .build();
+
+ return resourcesToStage.getPipelineResourcesDetector();
+}
+ }
+
+ class ClasspathScanningResourcesDetectorFactory
Review comment:
class comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 361107)
Time Spent: 14h 50m (was: 14h 40m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Fix For: 2.19.0
>
> Time Spent: 14h 50m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resour
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360390&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360390 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 17:28 Start Date: 16/Dec/19 17:28 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360390) Time Spent: 14.5h (was: 14h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Fix For: 2.19.0 > > Time Spent: 14.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360388 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 17:27 Start Date: 16/Dec/19 17:27 Worklog Time Spent: 10m Work Description: mxm commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566160168 Sounds good. Let's keep the alternative approach in the Git history then. Merging, tests were passing before and the force-push did not result in any changed files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360388) Time Spent: 14h 20m (was: 14h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 14h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360384&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360384 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 17:18 Start Date: 16/Dec/19 17:18 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566156575 I believe that it may be worth keeping the pr in history. It introduces another (valid) way of how we can do classpath scanning in java >=9 that I've seen in other projects. In case we have any problems with classgraph, we can revert to that version easily (as easy as `git revert 7e5a885` - the commit that introduces classgraph). I could still squash that - it will be a little bit tricky due to another commit relying on the changes but I think I can deal with the conflicts. Should I do this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360384) Time Spent: 14h 10m (was: 14h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 14h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360361 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 16:39 Start Date: 16/Dec/19 16:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566140540 Could you squash this commit? https://github.com/apache/beam/pull/10268/commits/74f20d82f51b802947314943108d8af4969fe0f2 Why would you keep an earlier version of this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360361) Time Spent: 14h (was: 13h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 14h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360346 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 16:20 Start Date: 16/Dec/19 16:20 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566132048 (squashed) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360346) Time Spent: 13h 50m (was: 13h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360344 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 16:19 Start Date: 16/Dec/19 16:19 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566132048 (rebased & squashed) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360344) Time Spent: 13h 40m (was: 13.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360332 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 16/Dec/19 15:46 Start Date: 16/Dec/19 15:46 Worklog Time Spent: 10m Work Description: mxm commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-566117095 Could you squash the relevant commits? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 360332) Time Spent: 13.5h (was: 13h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360305&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360305
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:44
Start Date: 16/Dec/19 14:44
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358254181
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesDetector.java
##
@@ -18,10 +18,10 @@
package org.apache.beam.runners.core.construction.resources;
import java.io.Serializable;
-import java.util.List;
+import java.util.stream.Stream;
/** Interface for an algorithm detecting classpath resources for pipelines. */
public interface PipelineResourcesDetector extends Serializable {
- List detect(ClassLoader classLoader);
+ Stream detect(ClassLoader classLoader);
Review comment:
Stream was a better fit in case we want to do filtering outside a
`PipelineResourcesDetector` implementation class. This is already done in
`PipelineResources.java`. This way, when a detector (even a 3rd party one,
provided via pipeline options) provides paths that we know we should not
accept, we can easily filter them out using stream's api and then materialize
the result to a collection of our choice (eg. List or Set - it's up to beam's
developers at this point).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360305)
Time Spent: 13h 20m (was: 13h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 13h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360300
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:31
Start Date: 16/Dec/19 14:31
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358264060
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/ClasspathScanningResourcesDetector.java
##
@@ -43,9 +43,9 @@ public ClasspathScanningResourcesDetector(ClassGraph
classGraph) {
* @return A list of absolute paths to the resources the class loader uses.
*/
@Override
- public List detect(ClassLoader classLoader) {
-List collect =
classGraph.addClassLoader(classLoader).getClasspathFiles();
+ public Stream detect(ClassLoader classLoader) {
Review comment:
(I responded in a suggestion below)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360300)
Time Spent: 13h 10m (was: 13h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 13h 10m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360299&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360299
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:31
Start Date: 16/Dec/19 14:31
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358264060
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/ClasspathScanningResourcesDetector.java
##
@@ -43,9 +43,9 @@ public ClasspathScanningResourcesDetector(ClassGraph
classGraph) {
* @return A list of absolute paths to the resources the class loader uses.
*/
@Override
- public List detect(ClassLoader classLoader) {
-List collect =
classGraph.addClassLoader(classLoader).getClasspathFiles();
+ public Stream detect(ClassLoader classLoader) {
Review comment:
(I responded in a below suggestion)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360299)
Time Spent: 13h (was: 12h 50m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 13h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360289&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360289
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:18
Start Date: 16/Dec/19 14:18
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358256436
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##
@@ -49,7 +50,19 @@
ClassLoader classLoader, PipelineOptions options) {
PipelineResourcesOptions artifactsRelatedOptions =
options.as(PipelineResourcesOptions.class);
-return
artifactsRelatedOptions.getPipelineResourcesDetector().detect(classLoader);
+return artifactsRelatedOptions
+.getPipelineResourcesDetector()
+.detect(classLoader)
+.filter(isStageable())
+.collect(Collectors.toList());
+ }
+
+ /**
+ * Returns a predicate for filtering all resources that are impossible to
stage (like gradle
+ * wrapper jars).
+ */
+ private static Predicate isStageable() {
+return resourcePath -> !resourcePath.contains("gradle/wrapper");
Review comment:
I tried to use classgraph's `blacklist*()` methods to filter these. However,
I couldn't get any of them working for this particular directory.
I still think that even despite that, filtering should also be possible (and
easy, that's why I used streams) outside detector's implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360289)
Time Spent: 12h 50m (was: 12h 40m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 12h 50m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360276&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360276
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:14
Start Date: 16/Dec/19 14:14
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358254181
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesDetector.java
##
@@ -18,10 +18,10 @@
package org.apache.beam.runners.core.construction.resources;
import java.io.Serializable;
-import java.util.List;
+import java.util.stream.Stream;
/** Interface for an algorithm detecting classpath resources for pipelines. */
public interface PipelineResourcesDetector extends Serializable {
- List detect(ClassLoader classLoader);
+ Stream detect(ClassLoader classLoader);
Review comment:
Stream was a better fit in case we want to do filtering outside a
`PipelineResourcesDetector` implementation class. This is already done in
`PipelineResources.java`. This way, when a detector (even a 3rd party one)
provides paths that we know we should not accept, we can easily filter them out
using stream's api and then materialize the result to a collection of our
choice (eg. List or Set - it's up to beam's developers at this point).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360276)
Time Spent: 12h 40m (was: 12.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 12h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360270&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360270
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:06
Start Date: 16/Dec/19 14:06
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358248161
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesDetector.java
##
@@ -18,10 +18,10 @@
package org.apache.beam.runners.core.construction.resources;
import java.io.Serializable;
-import java.util.List;
+import java.util.stream.Stream;
/** Interface for an algorithm detecting classpath resources for pipelines. */
public interface PipelineResourcesDetector extends Serializable {
- List detect(ClassLoader classLoader);
+ Stream detect(ClassLoader classLoader);
Review comment:
Would prefer `List`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360270)
Time Spent: 12h 20m (was: 12h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360271&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360271
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:06
Start Date: 16/Dec/19 14:06
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358249787
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/PipelineResources.java
##
@@ -49,7 +50,19 @@
ClassLoader classLoader, PipelineOptions options) {
PipelineResourcesOptions artifactsRelatedOptions =
options.as(PipelineResourcesOptions.class);
-return
artifactsRelatedOptions.getPipelineResourcesDetector().detect(classLoader);
+return artifactsRelatedOptions
+.getPipelineResourcesDetector()
+.detect(classLoader)
+.filter(isStageable())
+.collect(Collectors.toList());
+ }
+
+ /**
+ * Returns a predicate for filtering all resources that are impossible to
stage (like gradle
+ * wrapper jars).
+ */
+ private static Predicate isStageable() {
+return resourcePath -> !resourcePath.contains("gradle/wrapper");
Review comment:
Is there a more programmatic way to filter these out, e.g. to stop at the
Gradle wrapper's classloader?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360271)
Time Spent: 12.5h (was: 12h 20m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=360269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-360269
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 16/Dec/19 14:06
Start Date: 16/Dec/19 14:06
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r358247892
##
File path:
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources/ClasspathScanningResourcesDetector.java
##
@@ -43,9 +43,9 @@ public ClasspathScanningResourcesDetector(ClassGraph
classGraph) {
* @return A list of absolute paths to the resources the class loader uses.
*/
@Override
- public List detect(ClassLoader classLoader) {
-List collect =
classGraph.addClassLoader(classLoader).getClasspathFiles();
+ public Stream detect(ClassLoader classLoader) {
Review comment:
Could we leave this `List`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 360269)
Time Spent: 12h 20m (was: 12h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 12h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358926 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 20:41 Start Date: 12/Dec/19 20:41 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-565177732 @Ardagan and @mxm I posted another commit that prevents timeouts of "JavaPortabilityApi PreCommit" tests. I described the problem in the commit message. Previous commits remained unchanged. Could you take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358926) Time Spent: 12h 10m (was: 12h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358800 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 16:58 Start Date: 12/Dec/19 16:58 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-565092515 I think I know what the problem was - the error was due to the fact that classGraph returns .gradle/wrapper/* jars too. The previous algorithm (based only on urlClassLoader) couldn't do that. We're unable to stage wrapper's - the jars are already in use by ./gradlew process. Filtering out the jars helps. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358800) Time Spent: 12h (was: 11h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358799 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 16:58 Start Date: 12/Dec/19 16:58 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-565092515 I think I know what the problem was - the error was due to the fact that classGraph returns .gradle/wrapper/* jars too. The previous algorithm (based only on urlClassLoader) couldn't do that. We're unable to stage wrapper's - the jars are already in use by ./gradlew process. Filtering out the jars helps. :exploding_head: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358799) Time Spent: 11h 50m (was: 11h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358798&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358798 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 16:57 Start Date: 12/Dec/19 16:57 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-565092515 I think I know what the problem was - the error was due to the fact that classGraph returns .gradle/wrapper/* jars too. The previous algorithm (based only on urlClassLoader) couldn't do that. We're unable to stage wrapper's - the jars are already in use by ./gradlew process. Filtering out the jars helps. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358798) Time Spent: 11h 40m (was: 11.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358671 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 12:56 Start Date: 12/Dec/19 12:56 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564995651 It's `:runners:google-cloud-dataflow-java:examples:verifyFnApiWorker ` task. More specifically, tests: - org.apache.beam.examples.WordCountIT - org.apache.beam.examples.WindowedWordCountIT They hang and no data is inserted to the pipelines as far as I see. However, stackdriver says.. (screenshot) https://user-images.githubusercontent.com/1932045/70713754-f6639a80-1ce6-11ea-98f4-e5b414c73a1d.png";> https://user-images.githubusercontent.com/1932045/70713810-12ffd280-1ce7-11ea-9710-026f9081a370.png";> Still investigating... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358671) Time Spent: 11.5h (was: 11h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=358544&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-358544 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 12/Dec/19 10:39 Start Date: 12/Dec/19 10:39 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564951388 Looking at the JavaPortabilityApi PreCommit tests. It's weird that they are timeouting... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 358544) Time Spent: 11h 20m (was: 11h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357837&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357837 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 11/Dec/19 13:14 Start Date: 11/Dec/19 13:14 Worklog Time Spent: 10m Work Description: mxm commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564533492 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357837) Time Spent: 11h 10m (was: 11h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357717&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357717 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 11/Dec/19 09:10 Start Date: 11/Dec/19 09:10 Worklog Time Spent: 10m Work Description: mxm commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564448063 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357717) Time Spent: 11h (was: 10h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357244&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357244 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 17:13 Start Date: 10/Dec/19 17:13 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564133817 I couldn't reproduce the Jenkins behavior in the dockerized-jenkins clone of our environment. Short description of the problem - Jenkins does not try to run validatesRunner task as I specified in the jobDsl definition. It tries to use an old one - validatesJava11Runner even despite the fact that I retriggered seed job. I decided to improve the Java 11 Jenkins jobs in a separate pr in order not to block the PR (and be even more transparent about this by creating a separate ticket). Jira issue for the jobs improvement: https://issues.apache.org/jira/browse/BEAM-8940 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357244) Time Spent: 10h 50m (was: 10h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357241 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 17:11 Start Date: 10/Dec/19 17:11 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564133817 I couldn't reproduce the Jenkins behavior in the dockerized-jenkins clone of our environment. Short description of the problem - jenkins does not try to run validatesRunner task as I specified in the jobDsl definition. It tries to use an old one - validatesJava11Runner even despite the fact that I retriggered seed job. I decided to improve the Java 11 Jenkins jobs in a separate pr in order not to block the PR (and be even more transparent about this). Created an issue for this: https://issues.apache.org/jira/browse/BEAM-8940 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357241) Time Spent: 10h 40m (was: 10.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357240
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 10/Dec/19 17:10
Start Date: 10/Dec/19 17:10
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r356166362
##
File path:
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
##
@@ -19,9 +19,11 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
+final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
- 'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
+'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
Review comment:
I decided to remove the commit that introduced the change - I had some
problems with Jenkins not reloading the job definition. An issue for the
Jenkins jobs is created here: https://issues.apache.org/jira/browse/BEAM-8940
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 357240)
Time Spent: 10.5h (was: 10h 20m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357239 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 17:08 Start Date: 10/Dec/19 17:08 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564133817 I couldn't reproduce the Jenkins behavior in the dockerized-jenkins clone of our environment. Short description of the problem - jenkins does not try to run validatesRunner task as I specified in the jobDsl definition. It tries to use an old one - validatesJava11Runner even despite the fact that I retriggered seed job. I decided to improve the Java 11 Jenkins jobs in a separate pr in order not to block the PR. Created an issue for this: https://issues.apache.org/jira/browse/BEAM-8940 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357239) Time Spent: 10h 20m (was: 10h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 10h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357231 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 16:53 Start Date: 10/Dec/19 16:53 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564126742 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357231) Time Spent: 10h 10m (was: 10h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357105 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 14:30 Start Date: 10/Dec/19 14:30 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564058375 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357105) Time Spent: 10h (was: 9h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357100 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 14:16 Start Date: 10/Dec/19 14:16 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564033075 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357100) Time Spent: 9h 50m (was: 9h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357098 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 14:14 Start Date: 10/Dec/19 14:14 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564051272 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357098) Time Spent: 9h 40m (was: 9.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357072 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 13:31 Start Date: 10/Dec/19 13:31 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564033075 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357072) Time Spent: 9.5h (was: 9h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=357071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357071 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 10/Dec/19 13:31 Start Date: 10/Dec/19 13:31 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-564032989 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357071) Time Spent: 9h 20m (was: 9h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356262 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 15:11 Start Date: 09/Dec/19 15:11 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563283361 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356262) Time Spent: 9h 10m (was: 9h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356254 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 14:58 Start Date: 09/Dec/19 14:58 Worklog Time Spent: 10m Work Description: mwalenia commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563276681 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356254) Time Spent: 9h (was: 8h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356224 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 14:46 Start Date: 09/Dec/19 14:46 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563271277 I have to investigate more thoroughly why the Java11 VR Dataflow test fails. It tries to run a task which I don't use in this PR (validatesJava11Runner). @mwalenia maybe you could help? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356224) Time Spent: 8h 50m (was: 8h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356137&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356137 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 12:43 Start Date: 09/Dec/19 12:43 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563221356 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356137) Time Spent: 8h 40m (was: 8.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356118 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 12:16 Start Date: 09/Dec/19 12:16 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563211781 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356118) Time Spent: 8.5h (was: 8h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356094&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356094 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 11:26 Start Date: 09/Dec/19 11:26 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-563192990 @mxm @Ardagan thanks for the suggestions! I posted new changes + answered comments. Could you take another look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356094) Time Spent: 8h 20m (was: 8h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356091
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 09/Dec/19 11:25
Start Date: 09/Dec/19 11:25
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r355395768
##
File path:
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
##
@@ -19,9 +19,11 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
+final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
- 'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
+'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
Review comment:
LMK if you have any comments or in case I misunderstood. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356091)
Time Spent: 8h 10m (was: 8h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356090
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 09/Dec/19 11:23
Start Date: 09/Dec/19 11:23
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r355395027
##
File path: runners/core-construction-java/build.gradle
##
@@ -38,10 +38,12 @@ dependencies {
compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.vendored_grpc_1_21_0
compile library.java.vendored_guava_26_0_jre
+ compile library.java.classgraph
compile library.java.jackson_core
compile library.java.jackson_databind
compile library.java.joda_time
compile library.java.slf4j_api
+ testCompile project(path: ":sdks:java:core", configuration: "testRuntime")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356090)
Time Spent: 8h (was: 7h 50m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356087&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356087 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 09/Dec/19 11:20 Start Date: 09/Dec/19 11:20 Worklog Time Spent: 10m Work Description: lgajowy commented on pull request #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#discussion_r355393782 ## File path: .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy ## @@ -19,9 +19,11 @@ import CommonJobProperties as commonJobProperties import PostcommitJobBuilder +final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64' +final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64' Review comment: Created a separate issue for this: https://issues.apache.org/jira/browse/BEAM-8919 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 356087) Time Spent: 7h 50m (was: 7h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 7h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356075
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 09/Dec/19 11:05
Start Date: 09/Dec/19 11:05
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r355387523
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##
@@ -28,51 +31,30 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
/** Tests for PipelineResources. */
@RunWith(JUnit4.class)
public class PipelineResourcesTest {
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
- @Rule public transient ExpectedException thrown = ExpectedException.none();
@Test
- public void detectClassPathResourceWithFileResources() throws Exception {
+ public void testDetectsResourcesToStage() throws IOException {
File file = tmpFolder.newFile("file");
-File file2 = tmpFolder.newFile("file2");
-URLClassLoader classLoader =
-new URLClassLoader(new URL[] {file.toURI().toURL(),
file2.toURI().toURL()});
+URLClassLoader classLoader = new URLClassLoader(new URL[]
{file.toURI().toURL()});
+PipelineResourcesOptions options =
+PipelineOptionsFactory.create().as(PipelineResourcesOptions.class);
-assertEquals(
-ImmutableList.of(file.getAbsolutePath(), file2.getAbsolutePath()),
-PipelineResources.detectClassPathResourcesToStage(classLoader));
- }
-
- @Test
- public void detectClassPathResourcesWithUnsupportedClassLoader() {
-ClassLoader mockClassLoader = Mockito.mock(ClassLoader.class);
-thrown.expect(IllegalArgumentException.class);
-thrown.expectMessage("Unable to use ClassLoader to detect classpath
elements.");
+List detectedResources =
+PipelineResources.detectClassPathResourcesToStage(classLoader,
options);
-PipelineResources.detectClassPathResourcesToStage(mockClassLoader);
- }
-
- @Test
- public void detectClassPathResourceWithNonFileResources() throws Exception {
Review comment:
The resource that is not a jar/zip file or directory will be ignored - this
is the behavior of classgraph library. I moved the test to the
`ClasspathScanningResourcesDetectorTest.java` class and changed it accordingly
(but I forgot to change the name and I will do this in a followup commit 😶
...).
In case we still want to throw exceptions we'd need to detect this case on
our own rather than relying on classpath behaviour. Is this needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356075)
Time Spent: 7h 40m (was: 7.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=356064&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-356064
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 09/Dec/19 10:56
Start Date: 09/Dec/19 10:56
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r355382952
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##
@@ -28,51 +31,30 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
/** Tests for PipelineResources. */
@RunWith(JUnit4.class)
public class PipelineResourcesTest {
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
- @Rule public transient ExpectedException thrown = ExpectedException.none();
@Test
- public void detectClassPathResourceWithFileResources() throws Exception {
+ public void testDetectsResourcesToStage() throws IOException {
File file = tmpFolder.newFile("file");
-File file2 = tmpFolder.newFile("file2");
-URLClassLoader classLoader =
-new URLClassLoader(new URL[] {file.toURI().toURL(),
file2.toURI().toURL()});
+URLClassLoader classLoader = new URLClassLoader(new URL[]
{file.toURI().toURL()});
+PipelineResourcesOptions options =
+PipelineOptionsFactory.create().as(PipelineResourcesOptions.class);
-assertEquals(
-ImmutableList.of(file.getAbsolutePath(), file2.getAbsolutePath()),
-PipelineResources.detectClassPathResourcesToStage(classLoader));
- }
-
- @Test
- public void detectClassPathResourcesWithUnsupportedClassLoader() {
Review comment:
Classgraph will ignore the unsupported classloader (eg. a mocked one as in
this case) and will use other sources to detect resources, such as
java.class.path env property. I will add a test in
`ClasspathScanningResourcesDetectorTest.java` to double-check this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 356064)
Time Spent: 7.5h (was: 7h 20m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355293
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 06/Dec/19 16:10
Start Date: 06/Dec/19 16:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r354907960
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##
@@ -28,51 +31,30 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
/** Tests for PipelineResources. */
@RunWith(JUnit4.class)
public class PipelineResourcesTest {
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
- @Rule public transient ExpectedException thrown = ExpectedException.none();
@Test
- public void detectClassPathResourceWithFileResources() throws Exception {
+ public void testDetectsResourcesToStage() throws IOException {
File file = tmpFolder.newFile("file");
-File file2 = tmpFolder.newFile("file2");
-URLClassLoader classLoader =
-new URLClassLoader(new URL[] {file.toURI().toURL(),
file2.toURI().toURL()});
+URLClassLoader classLoader = new URLClassLoader(new URL[]
{file.toURI().toURL()});
+PipelineResourcesOptions options =
+PipelineOptionsFactory.create().as(PipelineResourcesOptions.class);
-assertEquals(
-ImmutableList.of(file.getAbsolutePath(), file2.getAbsolutePath()),
-PipelineResources.detectClassPathResourcesToStage(classLoader));
- }
-
- @Test
- public void detectClassPathResourcesWithUnsupportedClassLoader() {
-ClassLoader mockClassLoader = Mockito.mock(ClassLoader.class);
-thrown.expect(IllegalArgumentException.class);
-thrown.expectMessage("Unable to use ClassLoader to detect classpath
elements.");
+List detectedResources =
+PipelineResources.detectClassPathResourcesToStage(classLoader,
options);
-PipelineResources.detectClassPathResourcesToStage(mockClassLoader);
- }
-
- @Test
- public void detectClassPathResourceWithNonFileResources() throws Exception {
Review comment:
What about this case? How is it handled?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 355293)
Time Spent: 7h 20m (was: 7h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355292&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355292
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 06/Dec/19 16:10
Start Date: 06/Dec/19 16:10
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r354908133
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/resources/PipelineResourcesTest.java
##
@@ -28,51 +31,30 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.Mockito;
/** Tests for PipelineResources. */
@RunWith(JUnit4.class)
public class PipelineResourcesTest {
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
- @Rule public transient ExpectedException thrown = ExpectedException.none();
@Test
- public void detectClassPathResourceWithFileResources() throws Exception {
+ public void testDetectsResourcesToStage() throws IOException {
File file = tmpFolder.newFile("file");
-File file2 = tmpFolder.newFile("file2");
-URLClassLoader classLoader =
-new URLClassLoader(new URL[] {file.toURI().toURL(),
file2.toURI().toURL()});
+URLClassLoader classLoader = new URLClassLoader(new URL[]
{file.toURI().toURL()});
+PipelineResourcesOptions options =
+PipelineOptionsFactory.create().as(PipelineResourcesOptions.class);
-assertEquals(
-ImmutableList.of(file.getAbsolutePath(), file2.getAbsolutePath()),
-PipelineResources.detectClassPathResourcesToStage(classLoader));
- }
-
- @Test
- public void detectClassPathResourcesWithUnsupportedClassLoader() {
Review comment:
What about this case? How is it handled?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 355292)
Time Spent: 7h 20m (was: 7h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355262&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355262 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 15:26 Start Date: 06/Dec/19 15:26 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562616262 (I'm not sure if seed job works now...) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355262) Time Spent: 7h (was: 6h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355263&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355263 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 15:26 Start Date: 06/Dec/19 15:26 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562616262 (I'm not sure if seed job works now... looking) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355263) Time Spent: 7h 10m (was: 7h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355261 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 15:24 Start Date: 06/Dec/19 15:24 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562615343 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355261) Time Spent: 6h 50m (was: 6h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355253 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 15:08 Start Date: 06/Dec/19 15:08 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562608793 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355253) Time Spent: 6h 40m (was: 6.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355241 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:56 Start Date: 06/Dec/19 14:56 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562602899 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355241) Time Spent: 6.5h (was: 6h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355240 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:55 Start Date: 06/Dec/19 14:55 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562602575 @mxm could you take a look as well (we probably could use some "flink perspective" as well :) ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355240) Time Spent: 6h 20m (was: 6h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355239 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:55 Start Date: 06/Dec/19 14:55 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562602575 @mxm could you take a look as well (we probably could use some "flink perspective" :) ) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355239) Time Spent: 6h 10m (was: 6h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355231 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:42 Start Date: 06/Dec/19 14:42 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562597259 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355231) Time Spent: 6h (was: 5h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355210 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:19 Start Date: 06/Dec/19 14:19 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562589016 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355210) Time Spent: 5h 50m (was: 5h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=355209&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-355209 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 06/Dec/19 14:19 Start Date: 06/Dec/19 14:19 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562588935 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 355209) Time Spent: 5h 40m (was: 5.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354329&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354329 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 05/Dec/19 14:23 Start Date: 05/Dec/19 14:23 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562150319 The dataflow VR tests are timeouting: https://issues.apache.org/jira/browse/BEAM-8424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354329) Time Spent: 5.5h (was: 5h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354284 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 05/Dec/19 13:10 Start Date: 05/Dec/19 13:10 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562122391 Run JavaPortabilityApi PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354284) Time Spent: 5h 20m (was: 5h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354208&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354208 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 05/Dec/19 11:48 Start Date: 05/Dec/19 11:48 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562096039 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354208) Time Spent: 5h 10m (was: 5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354172&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354172 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 05/Dec/19 10:24 Start Date: 05/Dec/19 10:24 Worklog Time Spent: 10m Work Description: lgajowy commented on pull request #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#discussion_r354221674 ## File path: .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy ## @@ -19,9 +19,11 @@ import CommonJobProperties as commonJobProperties import PostcommitJobBuilder +final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64' +final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64' Review comment: Good point - I agree. This was copied from some other job ([VR_Direct_Java11](https://github.com/apache/beam/blob/fa37fc5e176e72fd346b9a0bb907d9726b33d018/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy)). If I'm not mistaken, adding new env variables to Jenkins has to go through asf infra so that they add the variables and then we can easily use them. Am I right @Ardagan ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354172) Time Spent: 5h (was: 4h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354168
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 05/Dec/19 10:10
Start Date: 05/Dec/19 10:10
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r354214465
##
File path:
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
##
@@ -19,9 +19,11 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
+final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
- 'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
+'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
Review comment:
In general, I think we should stick with using the builders to enforce a
common convention for all the jobs. Changing the convention and
improving/refactoring the builders (in case we want to do this) should be part
of different pr - this is already a huge one IMO and focuses on different
aspects (of course a jira ticket should be created prior this).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354168)
Time Spent: 4h 50m (was: 4h 40m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354167&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354167
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 05/Dec/19 10:05
Start Date: 05/Dec/19 10:05
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r354211593
##
File path:
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
##
@@ -19,9 +19,11 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
+final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
- 'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
+'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
Review comment:
I'm not sure if I understand you - we're using the
PostcommitJobBuilder.postCommitJob() method here. It creates 2 jobs:
- "beam_PostCommit_Java11_ValidatesRunner_Dataflow"
- "beam_PostCommit_Java11_ValidatesRunner_Dataflow_PR" (the suffix is added
automatically by the builder - there's no way to modify it if you're using the
builder)
What do you mean by "job title"? Do you mean the "name"? or the
"triggerPhrase" or "githubUIHint" in the
[postCommitJob](https://github.com/apache/beam/blob/master/.test-infra/jenkins/PostcommitJobBuilder.groovy#L47)
method? In your opinion, should I separate the creation of the two jobs and
have custom configurations (including naming) for the two?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 354167)
Time Spent: 4h 40m (was: 4.5h)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=354160&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354160 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 05/Dec/19 09:50 Start Date: 05/Dec/19 09:50 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-562053864 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354160) Time Spent: 4.5h (was: 4h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353579&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353579
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 04/Dec/19 16:56
Start Date: 04/Dec/19 16:56
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r353861585
##
File path: runners/core-construction-java/build.gradle
##
@@ -38,10 +38,12 @@ dependencies {
compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.vendored_grpc_1_21_0
compile library.java.vendored_guava_26_0_jre
+ compile library.java.classgraph
compile library.java.jackson_core
compile library.java.jackson_databind
compile library.java.joda_time
compile library.java.slf4j_api
+ testCompile project(path: ":sdks:java:core", configuration: "testRuntime")
Review comment:
Lets move this to the bottom along with another "testCompile project(...".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 353579)
Time Spent: 4h 20m (was: 4h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[
https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353578&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353578
]
ASF GitHub Bot logged work on BEAM-5495:
Author: ASF GitHub Bot
Created on: 04/Dec/19 16:56
Start Date: 04/Dec/19 16:56
Worklog Time Spent: 10m
Work Description: Ardagan commented on pull request #10268: [BEAM-5495]
PipelineResources algorithm is not working in most environments
URL: https://github.com/apache/beam/pull/10268#discussion_r353860351
##
File path:
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
##
@@ -19,9 +19,11 @@
import CommonJobProperties as commonJobProperties
import PostcommitJobBuilder
+final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
- 'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
+'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner
ValidatesRunner Tests On Java 11', this) {
Review comment:
Can we change job title to either:
a) trigger phrase
b) gradle target being executed.
Otherwise it doesn't give much additional information and loses a lot in
functionality.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 353578)
Time Spent: 4h 20m (was: 4h 10m)
> PipelineResources algorithm is not working in most environments
> ---
>
> Key: BEAM-5495
> URL: https://issues.apache.org/jira/browse/BEAM-5495
> Project: Beam
> Issue Type: Bug
> Components: runner-flink, runner-spark, sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Lukasz Gajowy
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Issue are:
> 1. it assumes the classloader is an URLClassLoader (not always true and java
> >= 9 breaks that as well for the app loader)
> 2. it uses loader.getURLs() which leads to including the JRE itself in the
> staged file
> Looks like this detect resource algorithm can't work and should be replaced
> by a SPI rather than a built-in and not extensible algorithm. Another valid
> alternative is to just drop that "guess" logic and force the user to set
> staged files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353577 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 16:56 Start Date: 04/Dec/19 16:56 Worklog Time Spent: 10m Work Description: Ardagan commented on pull request #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#discussion_r353859616 ## File path: .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy ## @@ -19,9 +19,11 @@ import CommonJobProperties as commonJobProperties import PostcommitJobBuilder +final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64' +final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64' Review comment: These should be loaded from env or be parameterized. Also these values should work for windows as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353577) Time Spent: 4h 10m (was: 4h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353553&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353553 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 16:14 Start Date: 04/Dec/19 16:14 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561716249 I think this is ready for review. I manually killed the website and python jobs so they do not use Jenkins resources needlessly (the changes surely do not involve python nor website). I also modified Java 11 validates runner Dataflow test and ran it so that it stages classpath resources with Java 11. More Java 11 tests will be fixed accordingly but in some following prs - this is already lot's of code to review. @lukecwik could you take a look? (CCing @kennknowles and @Ardagan too) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353553) Time Spent: 4h (was: 3h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353549&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353549 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 16:09 Start Date: 04/Dec/19 16:09 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561713823 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353549) Time Spent: 3h 50m (was: 3h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353537&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353537 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 15:45 Start Date: 04/Dec/19 15:45 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561703184 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353537) Time Spent: 3h 40m (was: 3.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353525&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353525 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 15:30 Start Date: 04/Dec/19 15:30 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561696668 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353525) Time Spent: 3.5h (was: 3h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353515 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 15:20 Start Date: 04/Dec/19 15:20 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561692393 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353515) Time Spent: 3h 20m (was: 3h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=353509&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353509 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 04/Dec/19 15:08 Start Date: 04/Dec/19 15:08 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561686916 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 353509) Time Spent: 3h 10m (was: 3h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352741&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352741 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 16:16 Start Date: 03/Dec/19 16:16 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561240499 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352741) Time Spent: 3h (was: 2h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352704&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352704 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 15:35 Start Date: 03/Dec/19 15:35 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561221873 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352704) Time Spent: 2h 50m (was: 2h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352701&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352701 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 15:31 Start Date: 03/Dec/19 15:31 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561220223 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352701) Time Spent: 2h 40m (was: 2.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352699 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 15:29 Start Date: 03/Dec/19 15:29 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561219247 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352699) Time Spent: 2.5h (was: 2h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352679&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352679 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:55 Start Date: 03/Dec/19 14:55 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561203900 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352679) Time Spent: 2h 20m (was: 2h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352678&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352678 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:54 Start Date: 03/Dec/19 14:54 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561203432 Run Dataflow ValidatesRunner Java 11 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352678) Time Spent: 2h 10m (was: 2h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352665&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352665 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:42 Start Date: 03/Dec/19 14:42 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197140 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352665) Time Spent: 1h 50m (was: 1h 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352666&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352666 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:42 Start Date: 03/Dec/19 14:42 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197089 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352666) Time Spent: 2h (was: 1h 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352664&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352664 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:41 Start Date: 03/Dec/19 14:41 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197668 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352664) Time Spent: 1h 40m (was: 1.5h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352660 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:40 Start Date: 03/Dec/19 14:40 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197089 Run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352660) Time Spent: 1h 10m (was: 1h) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352662 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:40 Start Date: 03/Dec/19 14:40 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197226 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352662) Time Spent: 1.5h (was: 1h 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352661 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 14:40 Start Date: 03/Dec/19 14:40 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561197140 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352661) Time Spent: 1h 20m (was: 1h 10m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352620&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352620 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 13:37 Start Date: 03/Dec/19 13:37 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561170913 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352620) Time Spent: 1h (was: 50m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352616&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352616 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 13:36 Start Date: 03/Dec/19 13:36 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561170429 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352616) Time Spent: 0.5h (was: 20m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5495) PipelineResources algorithm is not working in most environments
[ https://issues.apache.org/jira/browse/BEAM-5495?focusedWorklogId=352618&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352618 ] ASF GitHub Bot logged work on BEAM-5495: Author: ASF GitHub Bot Created on: 03/Dec/19 13:36 Start Date: 03/Dec/19 13:36 Worklog Time Spent: 10m Work Description: lgajowy commented on issue #10268: [BEAM-5495] PipelineResources algorithm is not working in most environments URL: https://github.com/apache/beam/pull/10268#issuecomment-561170577 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 352618) Time Spent: 50m (was: 40m) > PipelineResources algorithm is not working in most environments > --- > > Key: BEAM-5495 > URL: https://issues.apache.org/jira/browse/BEAM-5495 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Lukasz Gajowy >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Issue are: > 1. it assumes the classloader is an URLClassLoader (not always true and java > >= 9 breaks that as well for the app loader) > 2. it uses loader.getURLs() which leads to including the JRE itself in the > staged file > Looks like this detect resource algorithm can't work and should be replaced > by a SPI rather than a built-in and not extensible algorithm. Another valid > alternative is to just drop that "guess" logic and force the user to set > staged files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
