[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=101635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101635
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 14/May/18 08:59
Start Date: 14/May/18 08:59
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r187879854
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
##
@@ -69,216 +74,286 @@ private static MetricQueryResults
queryTestMetrics(PipelineResult result) {
.build());
}
- @Rule
- public final transient TestPipeline pipeline = TestPipeline.create();
-
- @Rule
- public final transient ExpectedException thrown = ExpectedException.none();
-
- @After
- public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
- }
-
- @Test
- public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
- }
-
- @Test
- public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
- }
-
- @Test
- public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
- }
-
- @Test
- public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+ /** Shared test helpers and setup/teardown. */
+ public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+ MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+ final Counter count = Metrics.counter(MetricsTest.class, "count");
+ final TupleTag output1 = new TupleTag(){};
+ final TupleTag output2 = new TupleTag(){};
+ pipeline
+ .apply(Create.of(5, 8, 13))
+ .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class,
"bundle");
+
+@StartBundle
+public void startBundle() {
+ bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values = Metrics.distribution(MetricsTest.class,
"input");
+ count.inc();
+ values.update(c.element());
+
+ c.output(c.element());
+ c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+ bundleDist.update(40L);
+}
+ }))
+ .apply("MyStep2", ParDo
+ .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values =
Metrics.distribution(MetricsTest.class, "input");
+ Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+ Integer element = c.element();
+ count.inc();
+ values.update(element);
+ gauge.set(12L);
+ c.output(element);
+ c.output(output2, element);
+}
+ })
+ .withOutputTags(output1, TupleTagList.of(output2)));
+ PipelineResult result = pipeline.run();
+
+ result.waitUntilFinish();
+ return result;
+}
}
- @Test
- public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+ /** Tests validating basic metric scenarios. */
+ @RunWith(JUnit4.class)
+ public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not fail even though there is no metrics container.
+ Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should n
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=99650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99650 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 08/May/18 17:47 Start Date: 08/May/18 17:47 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-387485718 Well, it seems to have sat for 4 days. So I think we could move forward and then could change those two test cases back if decided. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 99650) Time Spent: 6h 40m (was: 6.5h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=99570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99570 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 08/May/18 15:31 Start Date: 08/May/18 15:31 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-387443690 Ping @echauchot @kennknowles I believe all feedback has been addressed. Can you let me know if this is ready to merge? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 99570) Time Spent: 6.5h (was: 6h 20m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=99068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99068 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 07/May/18 15:33 Start Date: 07/May/18 15:33 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-387104433 Ping @echauchot; take a look at my comments above and let me know what you think. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 99068) Time Spent: 6h 20m (was: 6h 10m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98509&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98509
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 04/May/18 17:08
Start Date: 04/May/18 17:08
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r186147242
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
##
@@ -69,216 +74,286 @@ private static MetricQueryResults
queryTestMetrics(PipelineResult result) {
.build());
}
- @Rule
- public final transient TestPipeline pipeline = TestPipeline.create();
-
- @Rule
- public final transient ExpectedException thrown = ExpectedException.none();
-
- @After
- public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
- }
-
- @Test
- public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
- }
-
- @Test
- public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
- }
-
- @Test
- public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
- }
-
- @Test
- public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+ /** Shared test helpers and setup/teardown. */
+ public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+ MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+ final Counter count = Metrics.counter(MetricsTest.class, "count");
+ final TupleTag output1 = new TupleTag(){};
+ final TupleTag output2 = new TupleTag(){};
+ pipeline
+ .apply(Create.of(5, 8, 13))
+ .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class,
"bundle");
+
+@StartBundle
+public void startBundle() {
+ bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values = Metrics.distribution(MetricsTest.class,
"input");
+ count.inc();
+ values.update(c.element());
+
+ c.output(c.element());
+ c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+ bundleDist.update(40L);
+}
+ }))
+ .apply("MyStep2", ParDo
+ .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values =
Metrics.distribution(MetricsTest.class, "input");
+ Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+ Integer element = c.element();
+ count.inc();
+ values.update(element);
+ gauge.set(12L);
+ c.output(element);
+ c.output(output2, element);
+}
+ })
+ .withOutputTags(output1, TupleTagList.of(output2)));
+ PipelineResult result = pipeline.run();
+
+ result.waitUntilFinish();
+ return result;
+}
}
- @Test
- public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+ /** Tests validating basic metric scenarios. */
+ @RunWith(JUnit4.class)
+ public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not fail even though there is no metrics container.
+ Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not f
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98508
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 04/May/18 17:08
Start Date: 04/May/18 17:08
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r186146909
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
##
@@ -69,216 +74,286 @@ private static MetricQueryResults
queryTestMetrics(PipelineResult result) {
.build());
}
- @Rule
- public final transient TestPipeline pipeline = TestPipeline.create();
-
- @Rule
- public final transient ExpectedException thrown = ExpectedException.none();
-
- @After
- public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
- }
-
- @Test
- public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
- }
-
- @Test
- public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
- }
-
- @Test
- public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
- }
-
- @Test
- public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+ /** Shared test helpers and setup/teardown. */
+ public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+ MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+ final Counter count = Metrics.counter(MetricsTest.class, "count");
+ final TupleTag output1 = new TupleTag(){};
+ final TupleTag output2 = new TupleTag(){};
+ pipeline
+ .apply(Create.of(5, 8, 13))
+ .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class,
"bundle");
+
+@StartBundle
+public void startBundle() {
+ bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values = Metrics.distribution(MetricsTest.class,
"input");
+ count.inc();
+ values.update(c.element());
+
+ c.output(c.element());
+ c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+ bundleDist.update(40L);
+}
+ }))
+ .apply("MyStep2", ParDo
+ .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values =
Metrics.distribution(MetricsTest.class, "input");
+ Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+ Integer element = c.element();
+ count.inc();
+ values.update(element);
+ gauge.set(12L);
+ c.output(element);
+ c.output(output2, element);
+}
+ })
+ .withOutputTags(output1, TupleTagList.of(output2)));
+ PipelineResult result = pipeline.run();
+
+ result.waitUntilFinish();
+ return result;
+}
}
- @Test
- public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+ /** Tests validating basic metric scenarios. */
+ @RunWith(JUnit4.class)
+ public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not fail even though there is no metrics container.
+ Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not f
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98174&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98174
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 04/May/18 09:31
Start Date: 04/May/18 09:31
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r186030588
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
##
@@ -69,216 +74,286 @@ private static MetricQueryResults
queryTestMetrics(PipelineResult result) {
.build());
}
- @Rule
- public final transient TestPipeline pipeline = TestPipeline.create();
-
- @Rule
- public final transient ExpectedException thrown = ExpectedException.none();
-
- @After
- public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
- }
-
- @Test
- public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
- }
-
- @Test
- public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
- }
-
- @Test
- public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
- }
-
- @Test
- public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+ /** Shared test helpers and setup/teardown. */
+ public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+ MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+ final Counter count = Metrics.counter(MetricsTest.class, "count");
+ final TupleTag output1 = new TupleTag(){};
+ final TupleTag output2 = new TupleTag(){};
+ pipeline
+ .apply(Create.of(5, 8, 13))
+ .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class,
"bundle");
+
+@StartBundle
+public void startBundle() {
+ bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values = Metrics.distribution(MetricsTest.class,
"input");
+ count.inc();
+ values.update(c.element());
+
+ c.output(c.element());
+ c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+ bundleDist.update(40L);
+}
+ }))
+ .apply("MyStep2", ParDo
+ .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values =
Metrics.distribution(MetricsTest.class, "input");
+ Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+ Integer element = c.element();
+ count.inc();
+ values.update(element);
+ gauge.set(12L);
+ c.output(element);
+ c.output(output2, element);
+}
+ })
+ .withOutputTags(output1, TupleTagList.of(output2)));
+ PipelineResult result = pipeline.run();
+
+ result.waitUntilFinish();
+ return result;
+}
}
- @Test
- public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+ /** Tests validating basic metric scenarios. */
+ @RunWith(JUnit4.class)
+ public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not fail even though there is no metrics container.
+ Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98173&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98173
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 04/May/18 09:31
Start Date: 04/May/18 09:31
Worklog Time Spent: 10m
Work Description: echauchot commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r186030666
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
##
@@ -69,216 +74,286 @@ private static MetricQueryResults
queryTestMetrics(PipelineResult result) {
.build());
}
- @Rule
- public final transient TestPipeline pipeline = TestPipeline.create();
-
- @Rule
- public final transient ExpectedException thrown = ExpectedException.none();
-
- @After
- public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
- }
-
- @Test
- public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
- }
-
- @Test
- public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
- }
-
- @Test
- public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
- }
-
- @Test
- public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+ /** Shared test helpers and setup/teardown. */
+ public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+ MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+ final Counter count = Metrics.counter(MetricsTest.class, "count");
+ final TupleTag output1 = new TupleTag(){};
+ final TupleTag output2 = new TupleTag(){};
+ pipeline
+ .apply(Create.of(5, 8, 13))
+ .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class,
"bundle");
+
+@StartBundle
+public void startBundle() {
+ bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values = Metrics.distribution(MetricsTest.class,
"input");
+ count.inc();
+ values.update(c.element());
+
+ c.output(c.element());
+ c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+ bundleDist.update(40L);
+}
+ }))
+ .apply("MyStep2", ParDo
+ .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+ Distribution values =
Metrics.distribution(MetricsTest.class, "input");
+ Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+ Integer element = c.element();
+ count.inc();
+ values.update(element);
+ gauge.set(12L);
+ c.output(element);
+ c.output(output2, element);
+}
+ })
+ .withOutputTags(output1, TupleTagList.of(output2)));
+ PipelineResult result = pipeline.run();
+
+ result.waitUntilFinish();
+ return result;
+}
}
- @Test
- public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+ /** Tests validating basic metric scenarios. */
+ @RunWith(JUnit4.class)
+ public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not fail even though there is no metrics container.
+ Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+ assertNull(MetricsEnvironment.getCurrentContainer());
+ // Should not
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=98065&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-98065 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 03/May/18 22:43 Start Date: 03/May/18 22:43 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386458894 I think someone told me that the failure which you encountered is on `master` and fixed. Can you `git rebase -i` to resolve the fixups anyhow? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 98065) Time Spent: 5.5h (was: 5h 20m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97939 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 03/May/18 15:56 Start Date: 03/May/18 15:56 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386344774 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97939) Time Spent: 5h 20m (was: 5h 10m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97930 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 03/May/18 15:31 Start Date: 03/May/18 15:31 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386336631 FYI, current [Dataflow ValidatesRunner failure](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/186/) is due to the fact that current Gradle is set to max 2 workers. This is being addressed in https://github.com/apache/beam/pull/5218. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97930) Time Spent: 5h 10m (was: 5h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97699 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 02/May/18 21:26 Start Date: 02/May/18 21:26 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386126683 @kennknowles please take a look. Still waiting on Dataflow ValidatesRunner results, but if the current state is a clear improvement we should merge it since this test suite is currently executing at around 5 hours. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97699) Time Spent: 5h (was: 4h 50m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97626&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97626 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 02/May/18 18:22 Start Date: 02/May/18 18:22 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386073860 Java precommit now [fails on only a single test case](https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4838/), which is unrelated to this change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97626) Time Spent: 4h 40m (was: 4.5h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97628 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 02/May/18 18:23 Start Date: 02/May/18 18:23 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386073895 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97628) Time Spent: 4h 50m (was: 4h 40m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=97556&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-97556 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 02/May/18 16:10 Start Date: 02/May/18 16:10 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-386032850 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 97556) Time Spent: 4.5h (was: 4h 20m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=96235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96235 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 27/Apr/18 20:52 Start Date: 27/Apr/18 20:52 Worklog Time Spent: 10m Work Description: jasonkuster closed pull request #5240: [BEAM-4059] Bump Dataflow ValidatesRunner Timeout to 5h URL: https://github.com/apache/beam/pull/5240 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96235) Time Spent: 4h 20m (was: 4h 10m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=96223&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96223 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 27/Apr/18 20:28 Start Date: 27/Apr/18 20:28 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5240: [BEAM-4059] Bump Dataflow ValidatesRunner Timeout to 5h URL: https://github.com/apache/beam/pull/5240#issuecomment-385085452 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96223) Time Spent: 4h (was: 3h 50m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=96224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96224 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 27/Apr/18 20:28 Start Date: 27/Apr/18 20:28 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5240: [BEAM-4059] Bump Dataflow ValidatesRunner Timeout to 5h URL: https://github.com/apache/beam/pull/5240#issuecomment-385085467 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96224) Time Spent: 4h 10m (was: 4h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=96222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-96222 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 27/Apr/18 20:26 Start Date: 27/Apr/18 20:26 Worklog Time Spent: 10m Work Description: jasonkuster opened a new pull request #5240: [BEAM-4059] Bump Dataflow ValidatesRunner Timeout to 5h URL: https://github.com/apache/beam/pull/5240 R: @kennknowles @swegner Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Run `./gradlew build` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 96222) Time Spent: 3h 50m (was: 3h 40m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=94816&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94816 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 24/Apr/18 22:10 Start Date: 24/Apr/18 22:10 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-384096220 It seems the `@Enclosed` test runner doesn't like non-static inner classes. I'll work on converting new subclasses to static: https://scans.gradle.com/s/npfiidsczvefo/tests/failed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94816) Time Spent: 3h 40m (was: 3.5h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=94814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94814 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 24/Apr/18 22:07 Start Date: 24/Apr/18 22:07 Worklog Time Spent: 10m Work Description: kennknowles commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-384095631 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94814) Time Spent: 3.5h (was: 3h 20m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93383 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 19:03 Start Date: 20/Apr/18 19:03 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-383192267 Thanks for the quick review! All feedback is addressed; PTAL @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93383) Time Spent: 3h 20m (was: 3h 10m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93381 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 19:02 Start Date: 20/Apr/18 19:02 Worklog Time Spent: 10m Work Description: swegner commented on a change in pull request #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#discussion_r183142660 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java ## @@ -224,7 +224,7 @@ public UnserializableRecord decode( } @Test - @Category(ValidatesRunner.class) + @Category(NeedsRunner.class) Review comment: Ack, I'll leave as-is. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93381) Time Spent: 3h (was: 2h 50m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93379
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 19:02
Start Date: 20/Apr/18 19:02
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183142627
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
##
@@ -184,13 +184,13 @@ public void testSimpleCombineWithContext() {
}
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
Ok, I've reverted these changes and instead subdivided the tests into
subclasses.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93379)
Time Spent: 2h 50m (was: 2h 40m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93382
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 19:02
Start Date: 20/Apr/18 19:02
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183142712
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
##
@@ -138,7 +138,7 @@ public void testFlattenPCollectionsEmpty() {
}
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
Ack, reverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93382)
Time Spent: 3h 10m (was: 3h)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93378&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93378
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 19:01
Start Date: 20/Apr/18 19:01
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183142554
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
##
@@ -208,21 +208,21 @@ public String apply(final String input) {
@Rule
public final transient RuleChain chain =
RuleChain.outerRule(exception).around(pipeline);
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
Ack
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93378)
Time Spent: 2h 40m (was: 2.5h)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93358&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93358
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:50
Start Date: 20/Apr/18 18:50
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183139846
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##
@@ -173,7 +173,7 @@ public void testSuccessEncodedDecoded() throws IOException
{
* serializable.
*/
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
Reverted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93358)
Time Spent: 2.5h (was: 2h 20m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93357
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:50
Start Date: 20/Apr/18 18:50
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183139807
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
##
@@ -672,13 +672,13 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
@Test
- @Category({ValidatesRunner.class, UsesTestStream.class})
+ @Category({NeedsRunner.class, UsesTestStream.class})
Review comment:
I left a few that seemed like they may be exercising per-runner
functionality. I agree they probably shouldn't be necessary, but I'm not ready
to go through and validate that we have sufficient integration test coverage
before removing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93357)
Time Spent: 2h 20m (was: 2h 10m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93355
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:47
Start Date: 20/Apr/18 18:47
Worklog Time Spent: 10m
Work Description: swegner commented on a change in pull request #5193:
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183139179
##
File path: runners/direct-java/build.gradle
##
@@ -75,6 +76,8 @@ dependencies {
needsRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
needsRunner project(path: project.path, configuration: "shadow")
needsRunner project(path: project.path, configuration: "shadowTest")
+ validatesRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
Review comment:
Added a comment; the intention is to have a DirectRunner version of the
`@ValidatesRunner` suite so that I can run the same set of tests but in a
faster in-memory runner. It's not being used for Jenkins.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93355)
Time Spent: 2h 10m (was: 2h)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93348&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93348
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183132158
##
File path: runners/direct-java/build.gradle
##
@@ -75,6 +76,8 @@ dependencies {
needsRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
needsRunner project(path: project.path, configuration: "shadow")
needsRunner project(path: project.path, configuration: "shadowTest")
+ validatesRunner project(path: ":beam-sdks-java-core", configuration:
"shadowTest")
Review comment:
This will duplicate them, I think. Did you confirm? Because ValidatesRunner
extends NeedsRunner, all the VR tests should be pulled in by the other
execution. I think your version is cleaner, so I would agree to break this
relationship. We could do NR minus VR for the needsRunner gradle suite, or just
solve it with documentation/rename.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93348)
Time Spent: 1h 50m (was: 1h 40m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93347
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183132768
##
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
##
@@ -672,13 +672,13 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
@Test
- @Category({ValidatesRunner.class, UsesTestStream.class})
+ @Category({NeedsRunner.class, UsesTestStream.class})
Review comment:
+1 to this and all tests that are not one of the core primitives. It is true
that complex transforms sometimes exercise primitives in exciting ways, but we
can and should run integration tests separately from VR which are basically
compliance unit tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93347)
Time Spent: 1h 40m (was: 1.5h)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93350&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93350
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183133588
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/testing/TestPipelineTest.java
##
@@ -208,21 +208,21 @@ public String apply(final String input) {
@Rule
public final transient RuleChain chain =
RuleChain.outerRule(exception).around(pipeline);
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
This one is also iffy, maybe? But really these tests are also violating the
core tenet of tests which is straight-line readability. So I can't actually
tell at a glance what kind of test they really are. So go ahead and leave this
as you have it, I guess.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93350)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93346
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183133253
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
##
@@ -173,7 +173,7 @@ public void testSuccessEncodedDecoded() throws IOException
{
* serializable.
*/
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
I actually think that PAssertTest might be an exception - even though it is
a composite transform, it is important to run alongside the VR runner suite to
make sure the VR suite is meaningful. At least the particular pipelines that
make sure suites are vacuous.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93346)
Time Spent: 1.5h (was: 1h 20m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93349&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93349
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183133945
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
##
@@ -184,13 +184,13 @@ public void testSimpleCombineWithContext() {
}
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
For Combine there's an argument to bring some of them back since this is
almost always implemented as a primitive, even though it is not one,
technically. I would favor a JIRA TODO for factoring Combine into two suites -
one for the composite's various configs but one that is focused on testing a
runner's basic primitive implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93349)
Time Spent: 2h (was: 1h 50m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93345 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 18:29 Start Date: 20/Apr/18 18:29 Worklog Time Spent: 10m Work Description: kennknowles commented on a change in pull request #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#discussion_r183134308 ## File path: sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java ## @@ -224,7 +224,7 @@ public UnserializableRecord decode( } @Test - @Category(ValidatesRunner.class) + @Category(NeedsRunner.class) Review comment: How many runners are currently overriding Create as a primitive? If they aren't, then done! If they are, then (and this is a new thought that I thought of after my prior comments) another place for the tests to move is into ITs for just that runner, via some organization of modules. This is after-this-PR suggestion. I'm OK with this as-is unless runners are all turning it back into a primitive. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93345) Time Spent: 1.5h (was: 1h 20m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93344
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 20/Apr/18 18:29
Start Date: 20/Apr/18 18:29
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them
for better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r183134520
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
##
@@ -138,7 +138,7 @@ public void testFlattenPCollectionsEmpty() {
}
@Test
- @Category(ValidatesRunner.class)
+ @Category(NeedsRunner.class)
Review comment:
`Flatten.pCollections()` is a primitive that should be ValidatesRunner.
`Flatten.iterables()` is unrelated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 93344)
Time Spent: 1h 20m (was: 1h 10m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93303&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93303 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 17:30 Start Date: 20/Apr/18 17:30 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-383167321 R: @kennknowles This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93303) Time Spent: 1h 10m (was: 1h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93302 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 17:30 Start Date: 20/Apr/18 17:30 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-383167270 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93302) Time Spent: 1h (was: 50m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93285 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 17:05 Start Date: 20/Apr/18 17:05 Worklog Time Spent: 10m Work Description: swegner commented on issue #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193#issuecomment-383160347 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93285) Time Spent: 50m (was: 40m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=93284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-93284 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 20/Apr/18 17:04 Start Date: 20/Apr/18 17:04 Worklog Time Spent: 10m Work Description: swegner opened a new pull request #5193: [BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for better parallelization URL: https://github.com/apache/beam/pull/5193 Dataflow ValidatesRunner test suite has gotten obscenely slow due to the number of ValidatesRunner tests. There are currently over 250, and each one takes at least 3 minutes to run on Dataflow. Many of them don't actually need to be run on every worker, and this test converts them to `@NeedsRunner` tests instead. Gradle also parallelizes tests differently, parallelizing at the test class level rather than per test case. As a result, the largest test class will be a bottleneck for the overall execution. This PR splits up large `@ValidatesRunner` test classes into scenario-based subclasses. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [ ] Write a pull request description that is detailed enough to understand: - [ ] What the pull request does - [ ] Why it does it - [ ] How it does it - [ ] Why this approach - [ ] Each commit in the pull request should have a meaningful subject line and body. - [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 93284) Time Spent: 40m (was: 0.5h) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90639
]
ASF GitHub Bot logged work on BEAM-4059:
Author: ASF GitHub Bot
Created on: 12/Apr/18 23:58
Start Date: 12/Apr/18 23:58
Worklog Time Spent: 10m
Work Description: pabloem closed pull request #5116: [BEAM-4059]
Increasing timeout for Dataflow ValidatesRunner.
URL: https://github.com/apache/beam/pull/5116
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 07f3b6ca6c3..adb52114bc6 100644
---
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -25,8 +25,10 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') {
previousNames('beam_PostCommit_Java_ValidatesRunner_Dataflow')
previousNames('beam_PostCommit_Java_RunnableOnService_Dataflow')
- // Set common parameters. Sets a long (3 hour) timeout due to timeouts in
[BEAM-3775].
- common_job_properties.setTopLevelMainJobProperties(delegate, 'master', 180)
+ // Set common parameters.
+ // Sets a long (4 hour) timeout due to timeouts in BEAM-4059.
+ // TODO(BEAM-4059) - Speed up this test.
+ common_job_properties.setTopLevelMainJobProperties(delegate, 'master', 240)
// Publish all test results to Jenkins
publishers {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90639)
Time Spent: 0.5h (was: 20m)
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90589 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 12/Apr/18 21:05 Start Date: 12/Apr/18 21:05 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5116: [BEAM-4059] Increasing timeout for Dataflow ValidatesRunner. URL: https://github.com/apache/beam/pull/5116#issuecomment-380944043 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90589) Time Spent: 20m (was: 10m) > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90570 ] ASF GitHub Bot logged work on BEAM-4059: Author: ASF GitHub Bot Created on: 12/Apr/18 20:38 Start Date: 12/Apr/18 20:38 Worklog Time Spent: 10m Work Description: pabloem opened a new pull request #5116: [BEAM-4059] Increasing timeout for Dataflow ValidatesRunner. URL: https://github.com/apache/beam/pull/5116 This increases the timeout to 4 hours, while we can figure out why it's running so slowly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90570) Time Spent: 10m Remaining Estimate: 0h > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
