Hi Alex, to understand your long email. Lets say that when I'll go to release in June (0.9.9), I'll use instructions described in [1]. So that will create the sources needed to post. Then, to avouid later problems for people verifying I'll verify it with Maven and Ant (build with both, and test SDK generated in example apps. Then push to dist.a.o, create discuss and vote threads, and start the vote.
Is that ok? Thanks [1] https://github.com/apache/royale-asjs/wiki/New-Release-Manager El jue., 2 abr. 2020 a las 19:16, Alex Harui (<[email protected]>) escribió: > I'm not sure I understand the distinction. I think we want to do both. > The goal of code coverage is to try to exercise paths. We want to run "ant > release" because our Ant users might want to do that. And Ant does have > assertions AFAICT. It will report errors. Meanwhile, the standard for > the .tar.gz package is the one produced by "ant release" because that's the > recipe we've been using for years now. The Maven distribution's .tar.gz > has been shown to work in most cases, but AFAICT, is nearly as well tested > and has not been binary-compared. Ways to compare the two .tar.gz files > are needed and welcome. > > More and better tests are welcome. > > Do we have agreement? It sounds like it. So I will now be spending my > evenings on the release instead of writing lists of 100 things. > > -Alex > > On 4/2/20, 8:40 AM, "Yishay Weiss" <[email protected]> wrote: > > Chris, I think you’re missing Alex’s point. We’re not running Ant to > make sure it doesn’t blow up. We’re running it to make sure the resultant > tar.gz/zip files are identical to the ones created by Maven. Alex, please > correct me if I’m wrong. > > ________________________________ > From: Christofer Dutz <[email protected]> > Sent: Thursday, April 2, 2020 4:42:00 PM > To: [email protected] <[email protected]> > Subject: Re: Royale Releases > > And to add a little to that [1] > > "In computer science, test coverage is a measure used to describe the > degree to which the source code of a program is executed when a particular > test suite runs." > > So no test, no coverage. Just using something and it's not blowing up > isn't a test for me. It's better than nothing however. > > Chris > > > > [1] > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCode_coverage&data=02%7C01%7Caharui%40adobe.com%7Cd317247f5606472fc5b208d7d71c300d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214388378762332&sdata=emW5sFytkSu0CWSVicIrmHRA3SxiJqi26n%2BFrF%2FYX5E%3D&reserved=0 > > Am 02.04.20, 15:39 schrieb "Christofer Dutz" < > [email protected]>: > > Hi folks, > > I would say we have coverage for both maven ant equally as we're > building the same code. > > However we are missing the important assertions. It's not that the > Ant build is running some tests Maven isn't. > It's just that the settings for Ant seem to be different than for > Maven and the Ant ones happen to work. > > Ideally there would be real tests that test the output of both to > see if it works in both cases. > > Chris > > > > Am 02.04.20, 15:15 schrieb "Harbs" <[email protected]>: > > Adding more coverage for Maven is good. > > Removing coverage for Ant is not. > > Do you agree? > > > On Apr 2, 2020, at 4:07 PM, Carlos Rovira < > [email protected]> wrote: > > > > Hi Harbs, > > > > I think what we're trying to say is that until now we > released with Maven > > and Ant, and that was hiding a flaw in Maven (SVG example). > So that means > > what we were trying to cover was not covered clearly, so the > premise is not > > right. > > > > > > > > El jue., 2 abr. 2020 a las 14:56, Harbs (< > [email protected]>) escribió: > > > >> No one is arguing that we shouldn’t add more tests. > >> > >> Please let’s not make it seem like there’s a disagreement > about that. > >> > >>> On Apr 2, 2020, at 10:46 AM, Carlos Rovira < > [email protected]> > >> wrote: > >>> > >>> Hi Alex, > >>> > >>> first, many thanks for the detailed email. I'll comment on > this later as > >> I > >>> have more time. > >>> > >>> For now, to add up a recent example on what Chris > commented: If you all > >>> remember a week ago I was trying to use SVG Images in a > blog example that > >>> was published 2 days ago. Nobody tried SVG Images before > building with > >>> maven, I know that since maven was not properly configured > and using that > >>> component from Maven was failing with an RTE. Probably we > have more > >> things > >>> not working the same way when build from Maven and Ant, > and that's > >>> something that will need people using that code paths in > test > >> applications > >>> (or in their own apps) to see if things works properly. > >>> > >>> I was recently introduced to "examples-integrationtest" by > Chris, that I > >>> plan to use soon as I can. I think is a great idea, since > you get a > >> Firefox > >>> running test interface of the real use of some concrete > royale code. I > >>> think passed until now unnoticed by all of us, and seems a > powerful tool. > >>> There's already an example about FlexStore with some basic > assertions. > >>> > >>> Again, thanks, and will comment on the rest later > >>> > >>> Carlos > >>> > >>> > >>> > >>> > >>> El jue., 2 abr. 2020 a las 9:20, Christofer Dutz (< > >> [email protected]>) > >>> escribió: > >>> > >>>> Hi Alex, > >>>> > >>>> just a point you are bringing up: "Code coverage". > >>>> > >>>> I strongly dislike the idea of "asjs" effectively being > the test for the > >>>> compiler. The reasoning behind this is: yes you do get > more code > >> covered, > >>>> but only the happy-path (ideally) and even if things go > wrong, the end > >>>> results aren't tested. Did add a module to asjs years ago > >>>> ("examples-integrationtest") that deployed the examples > in a tomcat > >> server > >>>> then opens a Firefox browser and clicks through 2 of the > examples (I > >> added > >>>> two dummy tests as an example, but seems no one touched > this after me). > >> I > >>>> did this because I remember us working on asjs for weeks > without anyone > >>>> noticing the compiler wasn't producing runnable code ... > same with the > >>>> little unit-tests that are still run for every example, > that simply > >> check > >>>> if an output is generated, because we had a prolonged > period of time > >> where > >>>> we were all working on different parts, but for quite > some time the > >>>> application compilation just didn't output anything and > no one noticed. > >>>> > >>>> So coverage is nothing without assertions (my opinion) > ... ok ... it's > >>>> slightly better than no coverage, but not much, in my > opinion. > >>>> > >>>> I think in parallel to this release discussions I have > seen numerous > >>>> threads about someone doing something that broke > something for someone > >>>> else. This could be addressed by increasing coverage by > providing > >> explicit > >>>> tests. > >>>> > >>>> Coming back to the releases: > >>>> I have no objections, if you do a "release" locally and > automate the > >>>> validation on the CI server (Which effectively would be > your proposal > >> to do > >>>> the first 12 steps on local hardware and the 13th on the > CI server). I > >> even > >>>> think that's a good idea ... There could be one step for > building a > >> release > >>>> from a given "git tag" for every build system and generic > means to > >> compare > >>>> tar.gz and zips produced by any build system with that of > another > >> (ideally > >>>> with better output than just a plain "true/false"). This > would even > >> help to > >>>> iron out the last potentially existing bumps out of the > Maven > >> distribution. > >>>> > >>>> Chris > >>>> > >>>> > >>>> > >>>> Am 02.04.20, 07:59 schrieb "Alex Harui" > <[email protected]>: > >>>> > >>>> Hi, > >>>> > >>>> This is my attempt to explain what goes into a release > in hopes that > >>>> we can understand and agree on what our release process > is. It became > >>>> apparent in my reading of the wiki page with the new > Maven steps and in > >>>> talking with Harbs today that there are still many > misunderstandings > >> about > >>>> what we do to create a release. I don't generally like > writing > >>>> instructions in English because it is easy to be > ambiguous. All of the > >>>> steps that we use to create releases had been captured in > Ant scripts > >> in a > >>>> much more explicit way, IMO, but I took the time to write > them down in > >>>> English here: > >>>> > >> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Froyale-asjs%2Fwiki%2FTask-List-For-Royale-Releases&data=02%7C01%7Caharui%40adobe.com%7Cd317247f5606472fc5b208d7d71c300d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214388378762332&sdata=HgyvdeHLRqIp4iBCvEDyxpcqW9eOrwH%2BqKcmfphmIk4%3D&reserved=0 > >>>> > >>>> I did this quickly by scanning the CI steps, the new > Maven steps and > >>>> the Ant scripts used in prior releases so there could > certainly be > >> mistakes > >>>> and missed steps. If I did my math right, the RM for > 0.9.7 will have to > >>>> complete over 100 tasks (essentially, typing a > command-line 100 times). > >>>> Future RMs, when we don't have to release build-tools, > will have about > >> 92 > >>>> steps. And I did not include voter verification checks > the RM should > >> run > >>>> before opening a vote (verifying that the artifacts > download and match > >>>> their checksums, etc). As an RM, I run a bunch of tests > on the RC > >> before > >>>> sending out the vote. Maybe we should add those to the > task list. > >>>> > >>>> I think there has been confusion about the use of Ant > in the release > >>>> process. Because I was the RM for the first set of > FlexJS/Royale > >> releases, > >>>> and I'm a lazy person who hates typing at the command > line, I created > >> Ant > >>>> scripts to execute these 100 steps. But I agree that it > is not a > >>>> requirement that other RMs must use the Ant scripts for > these > >> commands. If > >>>> you are the RM and like typing, go ahead. > >>>> > >>>> Then we found out that other people couldn't get > through this task > >>>> list. I think the 3 people who tried were having trouble > with Maven > >>>> uploads and downloads. So what I did was put the first > 40 steps or so > >> into > >>>> Jenkins jobs. And by doing that, Piotr was able to > produce our last > >>>> release. And that also saves on manually typing > commands. But again, > >>>> going forward, the RM gets to choose how they want to > execute these > >> steps. > >>>> > >>>> If you scan the set of steps, you'll see that "ant" is > only in there > >>>> once. I believe the recent threads have been about this > single command > >> out > >>>> of the 100+ commands. This is why this has been so > frustrating to me. > >> I > >>>> believe there is a solid technical reason for that one > command: it > >> proves > >>>> that the build.xml files in the source packages can build > the .tar.gz > >> that > >>>> are useful to NPM and IDE users who use Ant and want to > test a change. > >>>> > >>>> I think of it as code-coverage. If we had > code-coverage tools, we > >>>> would ask that the RM complete as much of the automated > code-coverage > >>>> testing as possible before posting the release for a > vote. That one > >>>> command increases our code coverage by running the > build.xml files. We > >>>> should be always working to increase automated code > coverage in the RC. > >>>> Certainly for me as RM, I will gladly watch TV as the > automated tests > >> run > >>>> because a failed RC means going back through many of the > first 25 > >> commands > >>>> again and wastes other people's time. Each RC is more > emails to read > >> and > >>>> more time from the voters and testers. > >>>> > >>>> If there are other ways for the RM to get the same or > better code > >>>> coverage on the build.xml files before posting the RC, we > can discuss > >> those > >>>> options. > >>>> > >>>> I am hopeful we can all agree on these simple > principles: Strive for > >>>> better code coverage and fewer failed RCs. Royale's main > purpose is to > >>>> save other people time. Let's do that in creating > releases too. > >>>> > >>>> One issue that was brought up recently was whether it > is a good > >>>> decision to have the RM test all of the build platforms > we support. > >>>> Suppose we add some other build system support or more in > the future? > >>>> Again, the code coverage principle applies here, but > also, I would like > >> us > >>>> to retain feature parity, and I also hope for as few RCs > and votes as > >>>> possible. So instead of having separate > votes/features/release-dates > >> for > >>>> the Maven artifacts vs the Ant artifacts vs the > SomeFutureBuildTech > >>>> artifacts, I think we should have one vote and keep them > all in sync. > >> If > >>>> we do ever get around to monthly or bi-monthly releases, > I think > >> separate > >>>> build platform releases would be too much work. > >>>> > >>>> But consider this thought I just had today: the RM > doesn't really > >>>> have to choose to do all 100 commands on a local machine > or with Ant > >>>> scripts or do the first 40 via CI. The RM can actually > pick and choose > >>>> commands to run on the CI server. The CI Jenkins jobs > are not a > >>>> separate/alternative release process, they are just > another way of > >>>> executing the first 40 steps. Using CI jobs actually > requires > >> additional > >>>> command-line cut-and-paste to push commits on the CI > server and to sign > >> and > >>>> validate binaries locally, but that's the trade-off of > not having to > >>>> configure your machine to successfully run all of the > automated tests > >> and > >>>> build systems, and being able to run a command by filling > in the version > >>>> number and rc number and hitting the "ok" button instead > of making sure > >> you > >>>> got the whole command typed in correctly. > >>>> > >>>> So, an RM can run the first 25 steps locally, then go > the CI server > >>>> and run what is now Jenkins Job "Royale_Release_Step_013" > (no need to > >> run > >>>> the first 12) and it will run tasks 26 through 32, and if > it is > >> successful, > >>>> then the RM has proven code coverage of the build.xml > files. (If the > >>>> resulting tar.gz and zips are not posted, then the RM > should verify that > >>>> they match the ones from Maven distribution). I would > encourage RMs to > >>>> also use the CI jobs that generate the emails to make > sure the subject > >> and > >>>> content is correct and contains the usual instructions so > we have > >>>> consistency. Maybe someday there will be CI jobs to do > the last 60+ > >> steps > >>>> if that helps. We could add a Jenkins job that runs an > Ant build on RC > >>>> artifacts on dist.a.o as well. > >>>> > >>>> I would like you all to help maintain the list of 100 > steps and other > >>>> documents related to the release process, and improve the > CI jobs and > >> Ant > >>>> steps if it helps you be a more efficient RM. I am > hopeful that now > >> that I > >>>> have hopefully explained our release process better, that > we can see > >> that > >>>> these 100+ steps just have to be done in some way. The > RM can figure > >> out > >>>> what way works best for them, but they must get through > all of them. > >>>> > >>>> Thanks, > >>>> -Alex > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> -- > >>> Carlos Rovira > >>> > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Cd317247f5606472fc5b208d7d71c300d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214388378762332&sdata=GmfwWqH%2FI7uyN%2FKAhOytA3akoAelmvQbFb78f4oQjRg%3D&reserved=0 > >> > >> > > > > -- > > Carlos Rovira > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Cd317247f5606472fc5b208d7d71c300d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214388378762332&sdata=GmfwWqH%2FI7uyN%2FKAhOytA3akoAelmvQbFb78f4oQjRg%3D&reserved=0 > > > > > > > > -- Carlos Rovira http://about.me/carlosrovira
