Re: Plan for the current testing situation
Guenter Milde wrote: > What is the policy regarding change-tracking for Development.lyx? Not needed: It is not translated. If we want to know latzer who edited a certain part then we can use git. Georg
Re: Plan for the current testing situation
On 2015-11-11, Scott Kostyshak wrote: > On Tue, Nov 10, 2015 at 03:30:24PM +, Guenter Milde wrote: >> On 2015-11-10, Scott Kostyshak wrote: >> > On Mon, Nov 09, 2015 at 11:19:16AM +0100, Vincent van Ravesteijn wrote: >> >> And yes of course there is an implicit rule that when your commit breaks >> >> something or has problems whatsoever you are expected to fix it within a >> >> reasonable timeframe. >> This is, what I prefer. Could we make this an explicit rule: >>When your commit breaks something or has problems whatsoever you are >>expected to fix it within a reasonable timeframe. >>If it is not possible to solve the problems, revert the commit or move it >>to a "feature branch" (remember, branching is dead easy with Git). >>"Reasonable" depends on the problems involved and may range from 1 day to >>some weeks. > This seems to be the most popular opinion. Günter, if you do not here > any more comments, please feel free to put this in Development.lyx. What is the policy regarding change-tracking for Development.lyx? Günter
Re: Plan for the current testing situation
On Tue, Nov 10, 2015 at 03:30:24PM +, Guenter Milde wrote: > On 2015-11-10, Scott Kostyshak wrote: > > On Mon, Nov 09, 2015 at 11:19:16AM +0100, Vincent van Ravesteijn wrote: > >> And yes of course there is an implicit rule that when your commit breaks > >> something or has problems whatsoever you are expected to fix it within a > >> reasonable timeframe. > > This is, what I prefer. Could we make this an explicit rule: > >When your commit breaks something or has problems whatsoever you are >expected to fix it within a reasonable timeframe. > >If it is not possible to solve the problems, revert the commit or move it >to a "feature branch" (remember, branching is dead easy with Git). > >"Reasonable" depends on the problems involved and may range from 1 day to >some weeks. This seems to be the most popular opinion. Günter, if you do not here any more comments, please feel free to put this in Development.lyx. > One problem with the current testing situation is, that many failing test > are due to fixes that correct behaviour that was wrong before (like > reporting missing characters in the output document as an error). > > Next, this led to discovery of the use of a wrong encoding with Xe/LuaTeX > and TeX fonts - solving this brought more problems to light. > > I believe such "indirect" problems must be solved "collectively" - after a > consensus whether to revert the "discovering" commit (and live with the old > hidden bugs), temporarily invert affected tests or do some hacks to get a > clean state, or have a concerted effort to solve the basic problems. Agreed. But I don't think this is a common situation. Scott
Re: Plan for the current testing situation
On 2015-11-10, Scott Kostyshak wrote: > On Mon, Nov 09, 2015 at 11:19:16AM +0100, Vincent van Ravesteijn wrote: >> >> So my preferred policy would be: >> if a >> >> commit is found to have broken a test, either the situation is resolved >> within >> >> a day (i.e. the bug is fixed or the test is fixed) or the commit is >> reverted >> >> (and perhaps pushed to a separate remote branch). >> > >> > A small remark: imposing a 1-day rule seems ad hoc to me. Do we even have >> such rules for compilation warnings? For compilation failures? For the >> latter, I imagine that people want it solved ASAP, but this arises more out >> of social pressure, not an ad hoc rule. > Social pressure does not work with everyone. Also, some people do not > understand why certain things are so frustrating to others. ... >> Yes, the 1-day rule might lead to frustrated developers and increases the >> noise in master branch even more. > OK. How about a 1 week rule then? Or you would prefer no rule and just > deal with it case-by-case and implicit rule as you mention below and as > Guillaume refers to? >> And yes of course there is an implicit rule that when your commit breaks >> something or has problems whatsoever you are expected to fix it within a >> reasonable timeframe. This is, what I prefer. Could we make this an explicit rule: When your commit breaks something or has problems whatsoever you are expected to fix it within a reasonable timeframe. If it is not possible to solve the problems, revert the commit or move it to a "feature branch" (remember, branching is dead easy with Git). "Reasonable" depends on the problems involved and may range from 1 day to some weeks. One problem with the current testing situation is, that many failing test are due to fixes that correct behaviour that was wrong before (like reporting missing characters in the output document as an error). Next, this led to discovery of the use of a wrong encoding with Xe/LuaTeX and TeX fonts - solving this brought more problems to light. I believe such "indirect" problems must be solved "collectively" - after a consensus whether to revert the "discovering" commit (and live with the old hidden bugs), temporarily invert affected tests or do some hacks to get a clean state, or have a concerted effort to solve the basic problems. Günter
Re: Plan for the current testing situation
Note: quoting is not correct (probably because of replying on phone). On Mon, Nov 09, 2015 at 11:19:16AM +0100, Vincent van Ravesteijn wrote: > >> Ideally, I would like for commits that break tests to be on a separate > git > >> branch. Once the bugs exposed by a commit are fixed or the tests are > fixed, or > >> the .lyx files are fixed, then the branch could be merged to master. This > >> allows us to presere a "0 failing tests" situation on the master branch > so it > >> is extremely easy to catch regressions. So my preferred policy would be: > if a > >> commit is found to have broken a test, either the situation is resolved > within > >> a day (i.e. the bug is fixed or the test is fixed) or the commit is > reverted > >> (and perhaps pushed to a separate remote branch). > > > > > > A small remark: imposing a 1-day rule seems ad hoc to me. Do we even have > such rules for compilation warnings? For compilation failures? For the > latter, I imagine that people want it solved ASAP, but this arises more out > of social pressure, not an ad hoc rule. Social pressure does not work with everyone. Also, some people do not understand why certain things are so frustrating to others. > However I like the idea of having a > "candidate" remote branch, which would open up possibilities. If that's > really needed. > > > > Yes, the 1-day rule might lead to frustrated developers and increases the > noise in master branch even more. OK. How about a 1 week rule then? Or you would prefer no rule and just deal with it case-by-case and implicit rule as you mention below and as Guillaume refers to? > And yes of course there is an implicit rule that when your commit breaks > something or has problems whatsoever you are expected to fix it within a > reasonable timeframe. > > I already proposed to have a "staging" branch where commits are pushed to > first. If there are no breaking tests and no other comments they would be > merged to master after n days. The problems with such a construction are: > - devs are not eager to have to learn/understand/use an even more complex > git-o-cratic workflow, Yes this is indeed a real issue. Maybe in the future once everyone is more comfortable with git we can consider this. > - there will be merge conflicts, people have to make sure to indicate which > commit depends on which (or use feature branches which takes us back to the > first point), > - you unavoidably need some sort of maintainer to decide what gets merged > when and who resolves merge conflicts and is reasonably always present (or > to try to use some autocomplex scripting). Seems like we cannot do this at this point then. Scott
Re: Plan for the current testing situation
On Mon, Nov 09, 2015 at 09:13:37AM +, Guillaume Munch wrote: > Le 02/11/2015 03:41, Scott Kostyshak a écrit : > >Thanks to all of those participating in the discussions about tests. I have > >learned a lot the last couple of weeks. Thank you also to those who have > >tried > >to run the tests. This to me is a great step forward. I know that the export > >tests are sloppy cheap tests, but I appreciate that some of you agree that > >they > >do have use, at least until we have unit testing. The question now is, how > >can > >we get the most use out of them and how can we maximize their signal to noise > >ratio? > > Thank you for taking time to make a summary message. The messages about > tests were too many, so I could not follow the discussions. If you could > write another summary once the situation with tests is resolved it would be > very useful. Yes, I will try to remember to do this. Hopefully we can put everything in Development.lyx. It might take a bit for the test situation to stabilize though. Please feel free to ask for an update of the situation if I forget to give one.
Re: Plan for the current testing situation
>> Ideally, I would like for commits that break tests to be on a separate git >> branch. Once the bugs exposed by a commit are fixed or the tests are fixed, or >> the .lyx files are fixed, then the branch could be merged to master. This >> allows us to presere a "0 failing tests" situation on the master branch so it >> is extremely easy to catch regressions. So my preferred policy would be: if a >> commit is found to have broken a test, either the situation is resolved within >> a day (i.e. the bug is fixed or the test is fixed) or the commit is reverted >> (and perhaps pushed to a separate remote branch). > > > A small remark: imposing a 1-day rule seems ad hoc to me. Do we even have such rules for compilation warnings? For compilation failures? For the latter, I imagine that people want it solved ASAP, but this arises more out of social pressure, not an ad hoc rule. However I like the idea of having a "candidate" remote branch, which would open up possibilities. If that's really needed. > Yes, the 1-day rule might lead to frustrated developers and increases the noise in master branch even more. And yes of course there is an implicit rule that when your commit breaks something or has problems whatsoever you are expected to fix it within a reasonable timeframe. I already proposed to have a "staging" branch where commits are pushed to first. If there are no breaking tests and no other comments they would be merged to master after n days. The problems with such a construction are: - devs are not eager to have to learn/understand/use an even more complex git-o-cratic workflow, - there will be merge conflicts, people have to make sure to indicate which commit depends on which (or use feature branches which takes us back to the first point), - you unavoidably need some sort of maintainer to decide what gets merged when and who resolves merge conflicts and is reasonably always present (or to try to use some autocomplex scripting). Vincent
Re: Plan for the current testing situation
Le 02/11/2015 03:41, Scott Kostyshak a écrit : Thanks to all of those participating in the discussions about tests. I have learned a lot the last couple of weeks. Thank you also to those who have tried to run the tests. This to me is a great step forward. I know that the export tests are sloppy cheap tests, but I appreciate that some of you agree that they do have use, at least until we have unit testing. The question now is, how can we get the most use out of them and how can we maximize their signal to noise ratio? Thank you for taking time to make a summary message. The messages about tests were too many, so I could not follow the discussions. If you could write another summary once the situation with tests is resolved it would be very useful. Ideally, I would like for commits that break tests to be on a separate git branch. Once the bugs exposed by a commit are fixed or the tests are fixed, or the .lyx files are fixed, then the branch could be merged to master. This allows us to presere a "0 failing tests" situation on the master branch so it is extremely easy to catch regressions. So my preferred policy would be: if a commit is found to have broken a test, either the situation is resolved within a day (i.e. the bug is fixed or the test is fixed) or the commit is reverted (and perhaps pushed to a separate remote branch). A small remark: imposing a 1-day rule seems ad hoc to me. Do we even have such rules for compilation warnings? For compilation failures? For the latter, I imagine that people want it solved ASAP, but this arises more out of social pressure, not an ad hoc rule. However I like the idea of having a "candidate" remote branch, which would open up possibilities. If that's really needed. Guillaume
Re: Plan for the current testing situation
Guenter Milde wrote: > On 2015-11-02, Georg Baum wrote: > >> 1) Fix the first half of bug #9744. This is an easy and safe change > > Actually, "allowing parallel configuration of TeX and non-TeX fonts" is a > precondition for the proposed config value "automatic" for "use non-TeX > fonts": > > * Currently, the LyX file stores font configuration only for one font set, > either TeX-fonts or non-TeX-fonts, however > > * documents with non-default fonts usually require configuration for both > sets > to get a consistent look (e.g. choose Times/Helvetica/Courier lookalikes > or "Linux Libertine" fonts - be it 8-bit or Unicode encoded fonts). Thanks, now I finally understand what was meant by "parallel configuration". I always thought that it meant the LaTeX side, so that it should somehow be possible to use both (which I did not understand how that could work). Such a change would be easy as well. >> Günter, maybe you want to have a try yourself (if Scott agrees to do >> this before 2.2.0)? > > I am willing to help but cannot do this. I'll try to find some time. Georg
Re: Plan for the current testing situation
On 2015-11-02, Georg Baum wrote: > Scott Kostyshak wrote: >> So in summary, regarding the XeTeX + TeX fonts, I propose the above policy >> to start with; and if we find that there really is such a low signal to >> noise ratio then we can change our minds and ignore them. But we will >> never know what the signal to noise ratio is if we just ignore them now. > This does all make sense (including the snipped part). Nevertheless I'd like > to propose that we put some intermediate steps in: > 1) Fix the first half of bug #9744. This is an easy and safe change Actually, "allowing parallel configuration of TeX and non-TeX fonts" is a precondition for the proposed config value "automatic" for "use non-TeX fonts": * Currently, the LyX file stores font configuration only for one font set, either TeX-fonts or non-TeX-fonts, however * documents with non-default fonts usually require configuration for both sets to get a consistent look (e.g. choose Times/Helvetica/Courier lookalikes or "Linux Libertine" fonts - be it 8-bit or Unicode encoded fonts). * Documents using non-Latin scripts or "exotic" accented characters may even fail to compile with the default fonts (either TeX or non-TeX) due to our new "missing character" error. (E.g. the "Latin Modern" default for non-TeX fonts has no small Greek letters nor Cyrillic ones.) > which has the additional benefit to make most of Günters concerns about > the "View XeTeX" toolbar button much less important. Just using non-TeX fonts will lead to failures with, e.g., all Russian, Bulgarian, Serbian, Greek and Hebrew manuals! > Also, if we do not do it before 2.2.0, we cannot do it for 2.2.x (file > format change). This also regards both parts of the proposed change. > Günter, maybe you want to have a try yourself (if Scott agrees to do > this before 2.2.0)? I am willing to help but cannot do this. > 2) Set the new "automatic" value for "use non-TeX fonts" for all documents > that should work with XeTeX in principle > 3) Re-evaluate the test status and decide then whether some tests do still > need to be suspended or inverted. > My guess would be that after these three steps, the tests would be much more > usable, and that the tests that do then still fail would point to real > problems which should not be ignored. As said above, many documents that shall work with XeTeX "in principle" require font customization with non-TeX fonts. Maybe we need a "default non-TeX fonts mechanism" for non-Latin scripts. Günter
Re: Plan for the current testing situation
On Mon, Nov 02, 2015 at 09:26:07PM +0100, Kornel Benko wrote: > > It seems to me that the underlying causes of the XeTeX + TeX fonts tests > > will > > not be fixed anytime soon (and that this is OK) so we have two options if we > > want to clean up the test output. We can either invert the current test > > failures or ignore all XeTeX TeX font tests. Ignoring the tests means that > > they > > will not be run. Whether we invert or ignore basically depends on how low we > > think the signal to noise ratio is. I am open to both possibilities, > > depending > > on what others prefer. I do have a preference for inverting the current > > tests > > (and not ignoring). > > I offered a solution already. Test are going to 'revertedTests'. > Through the file 'suspendedTests' we can select which of them should be > suspended. > > That way we have > ctest -L export # to run all but suspended tests > ctest -L suspended # runs only suspended (which are also inverted) OK seems reasonable. Scott
Re: Plan for the current testing situation
On Mon, Nov 02, 2015 at 09:21:23PM +0100, Georg Baum wrote: > Scott Kostyshak wrote: > > > So in summary, regarding the XeTeX + TeX fonts, I propose the above policy > > to start with; and if we find that there really is such a low signal to > > noise ratio then we can change our minds and ignore them. But we will > > never know what the signal to noise ratio is if we just ignore them now. > > This does all make sense (including the snipped part). Nevertheless I'd like > to propose that we put some intermediate steps in: > > 1) Fix the first half of bug #9744. This is an easy and safe change which > has the additional benefit to make most of Günters concerns about the "View > XeTeX" toolbar button much less important. Also, if we do not do it before > 2.2.0, we cannot do it for 2.2.x (file format change). Günter, maybe you > want to have a try yourself (if Scott agrees to do this before 2.2.0)? I see. Seems like it would have a good benefit. Would indeed be nice to have in 2.2.0 beta. > 2) Set the new "automatic" value for "use non-TeX fonts" for all documents > that should work with XeTeX in principle > > 3) Re-evaluate the test status and decide then whether some tests do still > need to be suspended or inverted. > > My guess would be that after these three steps, the tests would be much more > usable, and that the tests that do then still fail would point to real > problems which should not be ignored. OK so we will hold of from inverting tests, it seems you are suggesting. I'm fine with that. Scott
Re: Plan for the current testing situation
Am Sonntag, 1. November 2015 um 22:41:39, schrieb Scott Kostyshak > Thanks to all of those participating in the discussions about tests. I have > learned a lot the last couple of weeks. Thank you also to those who have tried > to run the tests. This to me is a great step forward. I know that the export > tests are sloppy cheap tests, but I appreciate that some of you agree that > they > do have use, at least until we have unit testing. The question now is, how can > we get the most use out of them and how can we maximize their signal to noise > ratio? > > Ideally, I would like for commits that break tests to be on a separate git > branch. Once the bugs exposed by a commit are fixed or the tests are fixed, or > the .lyx files are fixed, then the branch could be merged to master. This > allows us to presere a "0 failing tests" situation on the master branch so it > is extremely easy to catch regressions. So my preferred policy would be: if a > commit is found to have broken a test, either the situation is resolved within > a day (i.e. the bug is fixed or the test is fixed) or the commit is reverted > (and perhaps pushed to a separate remote branch). The only way I think there > will be support for this policy is if the tests are not fragile. If tests > commonly break even though a commit does not introduce a real bug, then this > policy would just create a lot of pain without much use. I think that > non-fragile export tests are a real possibility, but some changes might be > needed to achieve this. +1 But as you said, 'ideally'. > Back from my dream world, the problem with the current situtation is that so > many tests are failing (200 or so) that it makes it difficult to see if there > are new regressions. Especially as we move forward in the release process, > catching new regressions is extremely important. I thus propose some temporary > measures to get a clean test output. I think that the tests that are failing > because of known bugs should be inverted. For example, the tests that are > failing because of the change from babel to polyglossia could be inverted and > marked with the reason why they are inverted, and referred to in a bug report. > Georg (who did not introduce the regressions but is kindly working on them) is > aware of the failing tests, and is also working on better tests that are > specific to testing language nesting, so in some sense the tests have already > done their job and we should move on for now. +1 > A source of fragile tests is the XeTeX tests with TeX fonts. Günter has > pointed > out that exporting using XeTeX and TeX fonts could go from succeeding to > failing because of an added package, addition of one non-ASCII character, > babel > language files, font problems, and other issues. Further, we should not expect > that many users are using XeTeX with TeX fonts because it is not common (there > is a separate discussion we are having on discouraging such a combination). > Thus, a good question is should we even have tests for this combination? IMHO yes. Even if they may be 'suspended'. At least even XeTeX may evolve and the tests are ready to wake up. > It seems to me that the underlying causes of the XeTeX + TeX fonts tests will > not be fixed anytime soon (and that this is OK) so we have two options if we > want to clean up the test output. We can either invert the current test > failures or ignore all XeTeX TeX font tests. Ignoring the tests means that > they > will not be run. Whether we invert or ignore basically depends on how low we > think the signal to noise ratio is. I am open to both possibilities, depending > on what others prefer. I do have a preference for inverting the current tests > (and not ignoring). I offered a solution already. Test are going to 'revertedTests'. Through the file 'suspendedTests' we can select which of them should be suspended. That way we have ctest -L export # to run all but suspended tests ctest -L suspended # runs only suspended (which are also inverted) > I'm not convinced that the signal to noise ratio is *so* > low. If there is a significant chance that these tests will catch bugs that > our > other tests should not, they should be kept. I thus propose to invert the > current failures (and specify the reason each is failing) and then to treat > future XeTeX TeX font failures as failures of fragile tests. In some sense, > fragile tests can be OK if we know they are fragile and treat them as such. We > could amend our policy to be such that if XeTeX + TeX font tests fail, the > author of the commit that broke them can simply invert them without having the > responsibility of fixing them. The advantage of still keeping them (and not > "ignoring" them) is that the author of the commit might realize "my commit > should not have broken these tests" and thus find a bug in the commit. For > example, many commits do not even intend to change LaTeX. You might think that > other tests would catch most bugs
Re: Plan for the current testing situation
Scott Kostyshak wrote: > So in summary, regarding the XeTeX + TeX fonts, I propose the above policy > to start with; and if we find that there really is such a low signal to > noise ratio then we can change our minds and ignore them. But we will > never know what the signal to noise ratio is if we just ignore them now. This does all make sense (including the snipped part). Nevertheless I'd like to propose that we put some intermediate steps in: 1) Fix the first half of bug #9744. This is an easy and safe change which has the additional benefit to make most of Günters concerns about the "View XeTeX" toolbar button much less important. Also, if we do not do it before 2.2.0, we cannot do it for 2.2.x (file format change). Günter, maybe you want to have a try yourself (if Scott agrees to do this before 2.2.0)? 2) Set the new "automatic" value for "use non-TeX fonts" for all documents that should work with XeTeX in principle 3) Re-evaluate the test status and decide then whether some tests do still need to be suspended or inverted. My guess would be that after these three steps, the tests would be much more usable, and that the tests that do then still fail would point to real problems which should not be ignored. Georg
