Re: [DISCUSS] is it time to make Windows tests gating?
Windows tests are now in the main pipeline. There are still a handful of intermittent failures, but generally I expect to see them mostly green. The increased visibility should help drive the process of shaking out any remaining issues. The shape of the main pipeline is now very tall, which most people don’t seem to mind, but if you feel strongly that the font is just too small, a demo of an alternative layout is at https://github.com/apache/geode/pull/3606 > On May 17, 2019, at 9:53 AM, Dan Smith wrote: > > You're awesome Jens! > > -Dan > > On Fri, May 17, 2019 at 9:48 AM Jens Deppe wrote: > >> I fixed the first two and started ignoring >> AvailablePortHelperIntegrationTest for Windows (Windows has a default >> active port range wildly different than Linux which makes this test very >> flaky, and since it's just testing test code it seemed reasonable to ignore >> it). >> >> LauncherMemberMXBeanIntegration test still needs some love and I can look >> at that since I've worked on it in the past. >> >> --Jens >> >> On Fri, May 17, 2019 at 9:33 AM Dan Smith wrote: >> >>> Looking at the metrics for the windows jobs, it looks like the windows >>> tests are mostly red due to a few specific tests. The acceptance and gfsh >>> distributed tests jobs seem to be ok, it's just the unit tests and >>> integration tests that have problems. It also looks like >>> ExportConfigCommandTest stopped failing recently. Anyone want to take the >>> credit? >>> >>> I'd be in favor of putting them in the main pipeline but not the PR >>> pipeline, since they take so long. Based on the metrics it looks like we >>> are pretty close to green with these jobs in terms of effort. Anyone want >>> to volunteer to fix the remaining failures (or see if they have been >> fixed >>> recently)? >>> >>> -Dan >>> >>> >>> >> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209 >>> >>> CreateDiskStoreCommandTest: 2 failures >>> ExportConfigCommandTest: 34 failures >>> >>> >>> >> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209 >>> >>> LauncherMemberMXBeanIntegrationTest: 4 failures >>> AvailablePortHelperIntegrationTest: 17 failures >>> >>> >>> >>> >>> On Thu, May 16, 2019 at 5:32 PM Owen Nichols >> wrote: >>> I’ve created a PR for this: https://github.com/apache/geode/pull/3597 > On May 16, 2019, at 3:06 PM, Blake Bender >> wrote: > > +1 this needs to happen. I hope that doesn't cause too much pain for >>> the > dev team, but the native client team has a hard requirement that all >>> our > stuff works properly on Windows at all times, and it can cause >> trouble >>> if > random builds of the server can break us on Windows. > > I would hesitate to run these per-commit if they're taking that long, >>> but > daily is a thing that can easily happen. > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt < >>> bschucha...@pivotal.io > > wrote: > >> big +1, as long as artifacts of failed runs can be downloaded >> >> On 5/15/19 6:28 PM, Owen Nichols wrote: >>> For a very long time we’ve had Windows tests in the main pipeline >> (hidden away, not in the default view), but the pipeline proceeds to >> publish regardless of whether Windows tests fail or even run at all. >>> >>> Now seems like a good time to review whether to: >>> a) treat Windows tests as first-class tests and prevent the >> pipeline >> from proceeding if any test fails on Windows >>> b) keep as-is >>> c) change Windows tests to trigger only once a week rather than on every >> commit, if they are going to remain "informational only" >>> >>> One disadvantage to making Windows tests gating is that they >>> currently >> take much longer to run (around 5 hours, vs 2 hours for Linux >> tests). >> >>> >>
Re: [DISCUSS] is it time to make Windows tests gating?
You're awesome Jens! -Dan On Fri, May 17, 2019 at 9:48 AM Jens Deppe wrote: > I fixed the first two and started ignoring > AvailablePortHelperIntegrationTest for Windows (Windows has a default > active port range wildly different than Linux which makes this test very > flaky, and since it's just testing test code it seemed reasonable to ignore > it). > > LauncherMemberMXBeanIntegration test still needs some love and I can look > at that since I've worked on it in the past. > > --Jens > > On Fri, May 17, 2019 at 9:33 AM Dan Smith wrote: > > > Looking at the metrics for the windows jobs, it looks like the windows > > tests are mostly red due to a few specific tests. The acceptance and gfsh > > distributed tests jobs seem to be ok, it's just the unit tests and > > integration tests that have problems. It also looks like > > ExportConfigCommandTest stopped failing recently. Anyone want to take the > > credit? > > > > I'd be in favor of putting them in the main pipeline but not the PR > > pipeline, since they take so long. Based on the metrics it looks like we > > are pretty close to green with these jobs in terms of effort. Anyone want > > to volunteer to fix the remaining failures (or see if they have been > fixed > > recently)? > > > > -Dan > > > > > > > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209 > > > > CreateDiskStoreCommandTest: 2 failures > > ExportConfigCommandTest: 34 failures > > > > > > > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209 > > > > LauncherMemberMXBeanIntegrationTest: 4 failures > > AvailablePortHelperIntegrationTest: 17 failures > > > > > > > > > > On Thu, May 16, 2019 at 5:32 PM Owen Nichols > wrote: > > > > > I’ve created a PR for this: https://github.com/apache/geode/pull/3597 > > > > > > > On May 16, 2019, at 3:06 PM, Blake Bender > wrote: > > > > > > > > +1 this needs to happen. I hope that doesn't cause too much pain for > > the > > > > dev team, but the native client team has a hard requirement that all > > our > > > > stuff works properly on Windows at all times, and it can cause > trouble > > if > > > > random builds of the server can break us on Windows. > > > > > > > > I would hesitate to run these per-commit if they're taking that long, > > but > > > > daily is a thing that can easily happen. > > > > > > > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt < > > bschucha...@pivotal.io > > > > > > > > wrote: > > > > > > > >> big +1, as long as artifacts of failed runs can be downloaded > > > >> > > > >> On 5/15/19 6:28 PM, Owen Nichols wrote: > > > >>> For a very long time we’ve had Windows tests in the main pipeline > > > >> (hidden away, not in the default view), but the pipeline proceeds to > > > >> publish regardless of whether Windows tests fail or even run at all. > > > >>> > > > >>> Now seems like a good time to review whether to: > > > >>> a) treat Windows tests as first-class tests and prevent the > pipeline > > > >> from proceeding if any test fails on Windows > > > >>> b) keep as-is > > > >>> c) change Windows tests to trigger only once a week rather than on > > > every > > > >> commit, if they are going to remain "informational only" > > > >>> > > > >>> One disadvantage to making Windows tests gating is that they > > currently > > > >> take much longer to run (around 5 hours, vs 2 hours for Linux > tests). > > > >> > > > > > > > > >
Re: [DISCUSS] is it time to make Windows tests gating?
I fixed the first two and started ignoring AvailablePortHelperIntegrationTest for Windows (Windows has a default active port range wildly different than Linux which makes this test very flaky, and since it's just testing test code it seemed reasonable to ignore it). LauncherMemberMXBeanIntegration test still needs some love and I can look at that since I've worked on it in the past. --Jens On Fri, May 17, 2019 at 9:33 AM Dan Smith wrote: > Looking at the metrics for the windows jobs, it looks like the windows > tests are mostly red due to a few specific tests. The acceptance and gfsh > distributed tests jobs seem to be ok, it's just the unit tests and > integration tests that have problems. It also looks like > ExportConfigCommandTest stopped failing recently. Anyone want to take the > credit? > > I'd be in favor of putting them in the main pipeline but not the PR > pipeline, since they take so long. Based on the metrics it looks like we > are pretty close to green with these jobs in terms of effort. Anyone want > to volunteer to fix the remaining failures (or see if they have been fixed > recently)? > > -Dan > > > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209 > > CreateDiskStoreCommandTest: 2 failures > ExportConfigCommandTest: 34 failures > > > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209 > > LauncherMemberMXBeanIntegrationTest: 4 failures > AvailablePortHelperIntegrationTest: 17 failures > > > > > On Thu, May 16, 2019 at 5:32 PM Owen Nichols wrote: > > > I’ve created a PR for this: https://github.com/apache/geode/pull/3597 > > > > > On May 16, 2019, at 3:06 PM, Blake Bender wrote: > > > > > > +1 this needs to happen. I hope that doesn't cause too much pain for > the > > > dev team, but the native client team has a hard requirement that all > our > > > stuff works properly on Windows at all times, and it can cause trouble > if > > > random builds of the server can break us on Windows. > > > > > > I would hesitate to run these per-commit if they're taking that long, > but > > > daily is a thing that can easily happen. > > > > > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt < > bschucha...@pivotal.io > > > > > > wrote: > > > > > >> big +1, as long as artifacts of failed runs can be downloaded > > >> > > >> On 5/15/19 6:28 PM, Owen Nichols wrote: > > >>> For a very long time we’ve had Windows tests in the main pipeline > > >> (hidden away, not in the default view), but the pipeline proceeds to > > >> publish regardless of whether Windows tests fail or even run at all. > > >>> > > >>> Now seems like a good time to review whether to: > > >>> a) treat Windows tests as first-class tests and prevent the pipeline > > >> from proceeding if any test fails on Windows > > >>> b) keep as-is > > >>> c) change Windows tests to trigger only once a week rather than on > > every > > >> commit, if they are going to remain "informational only" > > >>> > > >>> One disadvantage to making Windows tests gating is that they > currently > > >> take much longer to run (around 5 hours, vs 2 hours for Linux tests). > > >> > > > > >
Re: [DISCUSS] is it time to make Windows tests gating?
Looking at the metrics for the windows jobs, it looks like the windows tests are mostly red due to a few specific tests. The acceptance and gfsh distributed tests jobs seem to be ok, it's just the unit tests and integration tests that have problems. It also looks like ExportConfigCommandTest stopped failing recently. Anyone want to take the credit? I'd be in favor of putting them in the main pipeline but not the PR pipeline, since they take so long. Based on the metrics it looks like we are pretty close to green with these jobs in terms of effort. Anyone want to volunteer to fix the remaining failures (or see if they have been fixed recently)? -Dan https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209 CreateDiskStoreCommandTest: 2 failures ExportConfigCommandTest: 34 failures https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209 LauncherMemberMXBeanIntegrationTest: 4 failures AvailablePortHelperIntegrationTest: 17 failures On Thu, May 16, 2019 at 5:32 PM Owen Nichols wrote: > I’ve created a PR for this: https://github.com/apache/geode/pull/3597 > > > On May 16, 2019, at 3:06 PM, Blake Bender wrote: > > > > +1 this needs to happen. I hope that doesn't cause too much pain for the > > dev team, but the native client team has a hard requirement that all our > > stuff works properly on Windows at all times, and it can cause trouble if > > random builds of the server can break us on Windows. > > > > I would hesitate to run these per-commit if they're taking that long, but > > daily is a thing that can easily happen. > > > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt > > > wrote: > > > >> big +1, as long as artifacts of failed runs can be downloaded > >> > >> On 5/15/19 6:28 PM, Owen Nichols wrote: > >>> For a very long time we’ve had Windows tests in the main pipeline > >> (hidden away, not in the default view), but the pipeline proceeds to > >> publish regardless of whether Windows tests fail or even run at all. > >>> > >>> Now seems like a good time to review whether to: > >>> a) treat Windows tests as first-class tests and prevent the pipeline > >> from proceeding if any test fails on Windows > >>> b) keep as-is > >>> c) change Windows tests to trigger only once a week rather than on > every > >> commit, if they are going to remain "informational only" > >>> > >>> One disadvantage to making Windows tests gating is that they currently > >> take much longer to run (around 5 hours, vs 2 hours for Linux tests). > >> > >
Re: [DISCUSS] is it time to make Windows tests gating?
I’ve created a PR for this: https://github.com/apache/geode/pull/3597 > On May 16, 2019, at 3:06 PM, Blake Bender wrote: > > +1 this needs to happen. I hope that doesn't cause too much pain for the > dev team, but the native client team has a hard requirement that all our > stuff works properly on Windows at all times, and it can cause trouble if > random builds of the server can break us on Windows. > > I would hesitate to run these per-commit if they're taking that long, but > daily is a thing that can easily happen. > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt > wrote: > >> big +1, as long as artifacts of failed runs can be downloaded >> >> On 5/15/19 6:28 PM, Owen Nichols wrote: >>> For a very long time we’ve had Windows tests in the main pipeline >> (hidden away, not in the default view), but the pipeline proceeds to >> publish regardless of whether Windows tests fail or even run at all. >>> >>> Now seems like a good time to review whether to: >>> a) treat Windows tests as first-class tests and prevent the pipeline >> from proceeding if any test fails on Windows >>> b) keep as-is >>> c) change Windows tests to trigger only once a week rather than on every >> commit, if they are going to remain "informational only" >>> >>> One disadvantage to making Windows tests gating is that they currently >> take much longer to run (around 5 hours, vs 2 hours for Linux tests). >>
Re: [DISCUSS] is it time to make Windows tests gating?
+1 this needs to happen. I hope that doesn't cause too much pain for the dev team, but the native client team has a hard requirement that all our stuff works properly on Windows at all times, and it can cause trouble if random builds of the server can break us on Windows. I would hesitate to run these per-commit if they're taking that long, but daily is a thing that can easily happen. On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt wrote: > big +1, as long as artifacts of failed runs can be downloaded > > On 5/15/19 6:28 PM, Owen Nichols wrote: > > For a very long time we’ve had Windows tests in the main pipeline > (hidden away, not in the default view), but the pipeline proceeds to > publish regardless of whether Windows tests fail or even run at all. > > > > Now seems like a good time to review whether to: > > a) treat Windows tests as first-class tests and prevent the pipeline > from proceeding if any test fails on Windows > > b) keep as-is > > c) change Windows tests to trigger only once a week rather than on every > commit, if they are going to remain "informational only" > > > > One disadvantage to making Windows tests gating is that they currently > take much longer to run (around 5 hours, vs 2 hours for Linux tests). >
Re: [DISCUSS] is it time to make Windows tests gating?
big +1, as long as artifacts of failed runs can be downloaded On 5/15/19 6:28 PM, Owen Nichols wrote: For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all. Now seems like a good time to review whether to: a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows b) keep as-is c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only" One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).
Re: [DISCUSS] is it time to make Windows tests gating?
Agreed on making them gating if we want to say we run on Windows. If we don't want to say we run on Windows, delete the jobs. -michael On Thursday, May 16, 2019, Anilkumar Gingade wrote: > >> around 5 hours, vs 2 hours for Linux tests). > May be a good time to look at reducing/optimizing this. > > > On Thu, May 16, 2019 at 9:57 AM Ernest Burghardt > wrote: > > > Yes make them gating. > > Run them every commit, Windows is a supported platform. > > Red boxes get attention and Red boxes get fixed. > > > > EB > > > > On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer wrote: > > > > > I think we need to make sure our windows tests get to green... If we > > > make them gating then we will never release, but at the time be > > > motivated to fix them, in order to release. > > > > > > Maybe they run once every day... to at least start getting an idea of > > > health > > > > > > On 5/15/19 18:28, Owen Nichols wrote: > > > > For a very long time we’ve had Windows tests in the main pipeline > > > (hidden away, not in the default view), but the pipeline proceeds to > > > publish regardless of whether Windows tests fail or even run at all. > > > > > > > > Now seems like a good time to review whether to: > > > > a) treat Windows tests as first-class tests and prevent the pipeline > > > from proceeding if any test fails on Windows > > > > b) keep as-is > > > > c) change Windows tests to trigger only once a week rather than on > > every > > > commit, if they are going to remain "informational only" > > > > > > > > One disadvantage to making Windows tests gating is that they > currently > > > take much longer to run (around 5 hours, vs 2 hours for Linux tests). > > > > > >
Re: [DISCUSS] is it time to make Windows tests gating?
>> around 5 hours, vs 2 hours for Linux tests). May be a good time to look at reducing/optimizing this. On Thu, May 16, 2019 at 9:57 AM Ernest Burghardt wrote: > Yes make them gating. > Run them every commit, Windows is a supported platform. > Red boxes get attention and Red boxes get fixed. > > EB > > On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer wrote: > > > I think we need to make sure our windows tests get to green... If we > > make them gating then we will never release, but at the time be > > motivated to fix them, in order to release. > > > > Maybe they run once every day... to at least start getting an idea of > > health > > > > On 5/15/19 18:28, Owen Nichols wrote: > > > For a very long time we’ve had Windows tests in the main pipeline > > (hidden away, not in the default view), but the pipeline proceeds to > > publish regardless of whether Windows tests fail or even run at all. > > > > > > Now seems like a good time to review whether to: > > > a) treat Windows tests as first-class tests and prevent the pipeline > > from proceeding if any test fails on Windows > > > b) keep as-is > > > c) change Windows tests to trigger only once a week rather than on > every > > commit, if they are going to remain "informational only" > > > > > > One disadvantage to making Windows tests gating is that they currently > > take much longer to run (around 5 hours, vs 2 hours for Linux tests). > > >
Re: [DISCUSS] is it time to make Windows tests gating?
Yes make them gating. Run them every commit, Windows is a supported platform. Red boxes get attention and Red boxes get fixed. EB On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer wrote: > I think we need to make sure our windows tests get to green... If we > make them gating then we will never release, but at the time be > motivated to fix them, in order to release. > > Maybe they run once every day... to at least start getting an idea of > health > > On 5/15/19 18:28, Owen Nichols wrote: > > For a very long time we’ve had Windows tests in the main pipeline > (hidden away, not in the default view), but the pipeline proceeds to > publish regardless of whether Windows tests fail or even run at all. > > > > Now seems like a good time to review whether to: > > a) treat Windows tests as first-class tests and prevent the pipeline > from proceeding if any test fails on Windows > > b) keep as-is > > c) change Windows tests to trigger only once a week rather than on every > commit, if they are going to remain "informational only" > > > > One disadvantage to making Windows tests gating is that they currently > take much longer to run (around 5 hours, vs 2 hours for Linux tests). >
Re: [DISCUSS] is it time to make Windows tests gating?
I think we need to make sure our windows tests get to green... If we make them gating then we will never release, but at the time be motivated to fix them, in order to release. Maybe they run once every day... to at least start getting an idea of health On 5/15/19 18:28, Owen Nichols wrote: For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all. Now seems like a good time to review whether to: a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows b) keep as-is c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only" One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).
Re: [DISCUSS] is it time to make Windows tests gating?
They are also as red as the day is long... On Wed, May 15, 2019, 18:28 Owen Nichols wrote: > For a very long time we’ve had Windows tests in the main pipeline (hidden > away, not in the default view), but the pipeline proceeds to publish > regardless of whether Windows tests fail or even run at all. > > Now seems like a good time to review whether to: > a) treat Windows tests as first-class tests and prevent the pipeline from > proceeding if any test fails on Windows > b) keep as-is > c) change Windows tests to trigger only once a week rather than on every > commit, if they are going to remain "informational only" > > One disadvantage to making Windows tests gating is that they currently > take much longer to run (around 5 hours, vs 2 hours for Linux tests).
[DISCUSS] is it time to make Windows tests gating?
For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all. Now seems like a good time to review whether to: a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows b) keep as-is c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only" One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).