Re: Removing :beam-website:testWebsite from gradle build target
> it would be good to have a sort of weekly report on dead links Seeing as checking for broken external links returns a lot of false positives, I'd rather not spam everyone with them. However, I don't know if making it a postcommit will give it sufficient visibility. Not sure what the best way to handle this is--only thing I am sure of is I don't want this blocking PRs. Kyle Weaver ️ Software Engineer ️ github.com/ibzib ️ kcwea...@google.com ️ +1650203 On Tue, Apr 16, 2019 at 1:13 AM Ismaël Mejía wrote: > > +1 to removing link validation for website changes. However it would be good > to have a sort of weekly report on dead links or another alternative to be > aware of them. > > On Tue, Apr 16, 2019 at 2:43 AM Kyle Weaver wrote: >> >> I agree with Andrew that the external links checks are ultra-flaky and >> seldom strictly needed, so I filed a PR to make checking external links >> optional and disabled by default: https://github.com/apache/beam/pull/8318. >> Let me know what you all think. >> >> Kyle Weaver ️ Software Engineer ️ github.com/ibzib ️ kcwea...@google.com ️ >> +1650203 >> >> >> >> On Mon, Apr 1, 2019 at 11:05 AM Kenneth Knowles wrote: >>> >>> +1 thanks for noticing and raising yet another source of non-hermeticity >>> (plus the docker constraint) >>> >>> On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud wrote: +1 on this, particularly removing the dead link checker from default tests. It is effectively testing that ~20 random websites are up. I wonder if there is a way to limit it to locally testing links within the beam site? On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey wrote: > > Hi, > > after playing around with Gradle build for a while, I would like to > suggest to remove ':beam-website:testWebsite target from Gradle's check > task. > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on > the build task. E.g. A part time user is unable to execute these tests in > a docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > > Of course, these tests add lots of value and should definitely be > executed, but wouldn't it be sufficient, to run this task only dedicated, > i.e. by an explicit call to ':beam-website:testWebsite' o > ':websitePreCommit'? Any thoughts? > > best, > > michel > > [1] https://issues.apache.org/jira/browse/BEAM-6760
Re: Removing :beam-website:testWebsite from gradle build target
+1 to removing link validation for website changes. However it would be good to have a sort of weekly report on dead links or another alternative to be aware of them. On Tue, Apr 16, 2019 at 2:43 AM Kyle Weaver wrote: > I agree with Andrew that the external links checks are ultra-flaky and > seldom strictly needed, so I filed a PR to make checking external links > optional and disabled by default: https://github.com/apache/beam/pull/8318. > Let me know what you all think. > > Kyle Weaver ️ Software Engineer ️ github.com/ibzib ️ > kcwea...@google.com ️ +1650203 > > > On Mon, Apr 1, 2019 at 11:05 AM Kenneth Knowles wrote: > >> +1 thanks for noticing and raising yet another source of non-hermeticity >> (plus the docker constraint) >> >> On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud >> wrote: >> >>> +1 on this, particularly removing the dead link checker from default >>> tests. It is effectively testing that ~20 random websites are up. I wonder >>> if there is a way to limit it to locally testing links within the beam site? >>> >>> On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey >>> wrote: >>> Hi, after playing around with Gradle build for a while, I would like to suggest to remove ':beam-website:testWebsite target from Gradle's check task. Rationale: - the task seems to be very flaky. In fact, I always need to add '-x :beam-website:testWebsite' to my build [1] - task uses docker, which imho adds a (unnecessary) severe constraint on the build task. E.g. A part time user is unable to execute these tests in a docker environment - these tests are accessing production environment. So myself hitting the build several times an hour could be considered a DOS attack. Of course, these tests add lots of value and should definitely be executed, but wouldn't it be sufficient, to run this task only dedicated, i.e. by an explicit call to ':beam-website:testWebsite' o ':websitePreCommit'? Any thoughts? best, michel [1] https://issues.apache.org/jira/browse/BEAM-6760 >>>
Re: Removing :beam-website:testWebsite from gradle build target
I agree with Andrew that the external links checks are ultra-flaky and seldom strictly needed, so I filed a PR to make checking external links optional and disabled by default: https://github.com/apache/beam/pull/8318. Let me know what you all think. Kyle Weaver ️ Software Engineer ️ github.com/ibzib ️ kcwea...@google.com ️ +1650203 On Mon, Apr 1, 2019 at 11:05 AM Kenneth Knowles wrote: > +1 thanks for noticing and raising yet another source of non-hermeticity > (plus the docker constraint) > > On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud wrote: > >> +1 on this, particularly removing the dead link checker from default >> tests. It is effectively testing that ~20 random websites are up. I wonder >> if there is a way to limit it to locally testing links within the beam site? >> >> On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey >> wrote: >> >>> Hi, >>> >>> after playing around with Gradle build for a while, I would like to >>> suggest to remove ':beam-website:testWebsite target from Gradle's check >>> task. >>> >>> Rationale: >>> - the task seems to be very flaky. In fact, I always need to add '-x >>> :beam-website:testWebsite' to my build [1] >>> - task uses docker, which imho adds a (unnecessary) severe constraint on >>> the build task. E.g. A part time user is unable to execute these tests in a >>> docker environment >>> - these tests are accessing production environment. So myself hitting >>> the build several times an hour could be considered a DOS attack. >>> >>> Of course, these tests add lots of value and should definitely be >>> executed, but wouldn't it be sufficient, to run this task only dedicated, >>> i.e. by an explicit call to ':beam-website:testWebsite' o >>> ':websitePreCommit'? Any thoughts? >>> >>> best, >>> >>> michel >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-6760 >>> >>
Re: Removing :beam-website:testWebsite from gradle build target
+1 thanks for noticing and raising yet another source of non-hermeticity (plus the docker constraint) On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud wrote: > +1 on this, particularly removing the dead link checker from default > tests. It is effectively testing that ~20 random websites are up. I wonder > if there is a way to limit it to locally testing links within the beam site? > > On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey wrote: > >> Hi, >> >> after playing around with Gradle build for a while, I would like to >> suggest to remove ':beam-website:testWebsite target from Gradle's check >> task. >> >> Rationale: >> - the task seems to be very flaky. In fact, I always need to add '-x >> :beam-website:testWebsite' to my build [1] >> - task uses docker, which imho adds a (unnecessary) severe constraint on >> the build task. E.g. A part time user is unable to execute these tests in a >> docker environment >> - these tests are accessing production environment. So myself hitting the >> build several times an hour could be considered a DOS attack. >> >> Of course, these tests add lots of value and should definitely be >> executed, but wouldn't it be sufficient, to run this task only dedicated, >> i.e. by an explicit call to ':beam-website:testWebsite' o >> ':websitePreCommit'? Any thoughts? >> >> best, >> >> michel >> >> [1] https://issues.apache.org/jira/browse/BEAM-6760 >> >
Re: Removing :beam-website:testWebsite from gradle build target
+1 if possible, removing link checks would be nice too, if they are unreliable and there is a way to disable them. On Mon, Apr 1, 2019 at 10:33 AM Mikhail Gryzykhin wrote: > +1 on this. I'd prefer to have this as pre-commit only. > > On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud wrote: > >> +1 on this, particularly removing the dead link checker from default >> tests. It is effectively testing that ~20 random websites are up. I wonder >> if there is a way to limit it to locally testing links within the beam site? >> >> On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey >> wrote: >> >>> Hi, >>> >>> after playing around with Gradle build for a while, I would like to >>> suggest to remove ':beam-website:testWebsite target from Gradle's check >>> task. >>> >>> Rationale: >>> - the task seems to be very flaky. In fact, I always need to add '-x >>> :beam-website:testWebsite' to my build [1] >>> - task uses docker, which imho adds a (unnecessary) severe constraint on >>> the build task. E.g. A part time user is unable to execute these tests in a >>> docker environment >>> - these tests are accessing production environment. So myself hitting >>> the build several times an hour could be considered a DOS attack. >>> >>> Of course, these tests add lots of value and should definitely be >>> executed, but wouldn't it be sufficient, to run this task only dedicated, >>> i.e. by an explicit call to ':beam-website:testWebsite' o >>> ':websitePreCommit'? Any thoughts? >>> >>> best, >>> >>> michel >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-6760 >>> >>
Re: Removing :beam-website:testWebsite from gradle build target
+1 on this. I'd prefer to have this as pre-commit only. On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud wrote: > +1 on this, particularly removing the dead link checker from default > tests. It is effectively testing that ~20 random websites are up. I wonder > if there is a way to limit it to locally testing links within the beam site? > > On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey wrote: > >> Hi, >> >> after playing around with Gradle build for a while, I would like to >> suggest to remove ':beam-website:testWebsite target from Gradle's check >> task. >> >> Rationale: >> - the task seems to be very flaky. In fact, I always need to add '-x >> :beam-website:testWebsite' to my build [1] >> - task uses docker, which imho adds a (unnecessary) severe constraint on >> the build task. E.g. A part time user is unable to execute these tests in a >> docker environment >> - these tests are accessing production environment. So myself hitting the >> build several times an hour could be considered a DOS attack. >> >> Of course, these tests add lots of value and should definitely be >> executed, but wouldn't it be sufficient, to run this task only dedicated, >> i.e. by an explicit call to ':beam-website:testWebsite' o >> ':websitePreCommit'? Any thoughts? >> >> best, >> >> michel >> >> [1] https://issues.apache.org/jira/browse/BEAM-6760 >> >
Re: Removing :beam-website:testWebsite from gradle build target
+1 on this, particularly removing the dead link checker from default tests. It is effectively testing that ~20 random websites are up. I wonder if there is a way to limit it to locally testing links within the beam site? On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey wrote: > Hi, > > after playing around with Gradle build for a while, I would like to > suggest to remove ':beam-website:testWebsite target from Gradle's check > task. > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on > the build task. E.g. A part time user is unable to execute these tests in a > docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > > Of course, these tests add lots of value and should definitely be > executed, but wouldn't it be sufficient, to run this task only dedicated, > i.e. by an explicit call to ':beam-website:testWebsite' o > ':websitePreCommit'? Any thoughts? > > best, > > michel > > [1] https://issues.apache.org/jira/browse/BEAM-6760 >
