[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489225#comment-16489225
]
Scott Wegner commented on BEAM-4059:
This was fixed with the work to prune {{@ValidatesRunner}} tests
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Scott Wegner
>Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451335#comment-16451335 ] Scott Wegner commented on BEAM-4059: These are still timing out a 3 hours. There are two changes which should help unblock: * [pr/4193|https://github.com/apache/beam/pull/5193]: Reduce number of ValidatesRunner tests and reorganize them for better parallelization * [pr/4218|https://github.com/apache/beam/pull/5218]: Remove Gradle worker cap. > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Scott Wegner >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437750#comment-16437750
]
Pablo Estrada commented on BEAM-4059:
-
In my test runs with higher number of parallel forks, I{m seeing that around
207 jobs, gradle is getting stuck, and executing new tests very slowly.
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437583#comment-16437583
]
Pablo Estrada commented on BEAM-4059:
-
Looking at the latest passing test, I've grouped the number of jobs started by
the minute they were started. In the latest passing test, at times 10 jobs were
started in the same minute. I'd think this means that the Maven-based tests
were faster due to the fact that they had more jobs running at the same time.
This would point towards increasing the no. of parallel forks in the Gradle
suite:
{{# of Tests, Time, Action
6 2018-04-05_19_16 started job
4 2018-04-05_19_19 started job
4 2018-04-05_19_20 started job
1 2018-04-05_19_22 started job
5 2018-04-05_19_23 started job
8 2018-04-05_19_26 started job
1 2018-04-05_19_29 started job
8 2018-04-05_19_30 started job
4 2018-04-05_19_33 started job
2 2018-04-05_19_36 started job
8 2018-04-05_19_37 started job
2 2018-04-05_19_39 started job
10 2018-04-05_19_40 started job
7 2018-04-05_19_43 started job
5 2018-04-05_19_44 started job
2 2018-04-05_19_46 started job
5 2018-04-05_19_47 started job
2 2018-04-05_19_48 started job
1 2018-04-05_19_49 started job
7 2018-04-05_19_50 started job
1 2018-04-05_19_51 started job
7 2018-04-05_19_53 started job
1 2018-04-05_19_54 started job
1 2018-04-05_19_55 started job
1 2018-04-05_19_56 started job
2 2018-04-05_19_57 started job
6 2018-04-05_19_58 started job
1 2018-04-05_20_00 started job
4 2018-04-05_20_01 started job
5 2018-04-05_20_02 started job
2 2018-04-05_20_03 started job
1 2018-04-05_20_04 started job
8 2018-04-05_20_05 started job
2 2018-04-05_20_06 started job
2 2018-04-05_20_07 started job
5 2018-04-05_20_08 started job
6 2018-04-05_20_09 started job
2 2018-04-05_20_10 started job
2 2018-04-05_20_11 started job
6 2018-04-05_20_12 started job
3 2018-04-05_20_13 started job
3 2018-04-05_20_14 started job
5 2018-04-05_20_15 started job
2 2018-04-05_20_16 started job
3 2018-04-05_20_17 started job
4 2018-04-05_20_18 started job
5 2018-04-05_20_19 started job
2 2018-04-05_20_20 started job
3 2018-04-05_20_21 started job
6 2018-04-05_20_22 started job
1 2018-04-05_20_23 started job
3 2018-04-05_20_24 started job
6 2018-04-05_20_25 started job
4 2018-04-05_20_26 started job
2 2018-04-05_20_27 started job
1 2018-04-05_20_28 started job
9 2018-04-05_20_29 started job
3 2018-04-05_20_30 started job
10 2018-04-05_20_32 started job
2 2018-04-05_20_33 started job
2 2018-04-05_20_34 started job
4 2018-04-05_20_35 started job
4 2018-04-05_20_36 started job
3 2018-04-05_20_37 started job
2 2018-04-05_20_38 started job
6 2018-04-05_20_39 started job
3 2018-04-05_20_40 started job
9 2018-04-05_20_42 started job
3 2018-04-05_20_44 started job
6 2018-04-05_20_45 started job
3 2018-04-05_20_46 started job
2 2018-04-05_20_47 started job
1 2018-04-05_20_48 started job
1 2018-04-05_20_49 started job}}
> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436260#comment-16436260 ] Pablo Estrada commented on BEAM-4059: - I think for now the best thing is to increase the timeout, to get the tests to pass, and later work on improving the speed of the test. With that in mind, I will increase it to 4 hours, and add a todo pointing to this. > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436163#comment-16436163 ] Pablo Estrada commented on BEAM-4059: - Issue tracking same problem, back when Maven timeout was increased: https://issues.apache.org/jira/browse/BEAM-3775 > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436158#comment-16436158 ] Pablo Estrada commented on BEAM-4059: - The previous Maven Build ran 283 dataflow jobs, and ran in about 2 hours: [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/5290/consoleFull] It doesn't seem like Jenkins is trying to build many more than that number of jobs, but still failed. Perhaps Maven has more executors? > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle
[ https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436101#comment-16436101 ] Pablo Estrada commented on BEAM-4059: - The test used to take 2 hours while running in Maven. I'm looking into why it has started taking over 3 hours - and thus timing out - in Gradle. > Make sure Dataflow ValidatesRunner tests pass in Gradle > --- > > Key: BEAM-4059 > URL: https://issues.apache.org/jira/browse/BEAM-4059 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > > See > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
