No one is arguing that we shouldn’t add more tests. Please let’s not make it seem like there’s a disagreement about that.
> On Apr 2, 2020, at 10:46 AM, Carlos Rovira <[email protected]> wrote: > > Hi Alex, > > first, many thanks for the detailed email. I'll comment on this later as I > have more time. > > For now, to add up a recent example on what Chris commented: If you all > remember a week ago I was trying to use SVG Images in a blog example that > was published 2 days ago. Nobody tried SVG Images before building with > maven, I know that since maven was not properly configured and using that > component from Maven was failing with an RTE. Probably we have more things > not working the same way when build from Maven and Ant, and that's > something that will need people using that code paths in test applications > (or in their own apps) to see if things works properly. > > I was recently introduced to "examples-integrationtest" by Chris, that I > plan to use soon as I can. I think is a great idea, since you get a Firefox > running test interface of the real use of some concrete royale code. I > think passed until now unnoticed by all of us, and seems a powerful tool. > There's already an example about FlexStore with some basic assertions. > > Again, thanks, and will comment on the rest later > > Carlos > > > > > El jue., 2 abr. 2020 a las 9:20, Christofer Dutz (<[email protected]>) > escribió: > >> Hi Alex, >> >> just a point you are bringing up: "Code coverage". >> >> I strongly dislike the idea of "asjs" effectively being the test for the >> compiler. The reasoning behind this is: yes you do get more code covered, >> but only the happy-path (ideally) and even if things go wrong, the end >> results aren't tested. Did add a module to asjs years ago >> ("examples-integrationtest") that deployed the examples in a tomcat server >> then opens a Firefox browser and clicks through 2 of the examples (I added >> two dummy tests as an example, but seems no one touched this after me). I >> did this because I remember us working on asjs for weeks without anyone >> noticing the compiler wasn't producing runnable code ... same with the >> little unit-tests that are still run for every example, that simply check >> if an output is generated, because we had a prolonged period of time where >> we were all working on different parts, but for quite some time the >> application compilation just didn't output anything and no one noticed. >> >> So coverage is nothing without assertions (my opinion) ... ok ... it's >> slightly better than no coverage, but not much, in my opinion. >> >> I think in parallel to this release discussions I have seen numerous >> threads about someone doing something that broke something for someone >> else. This could be addressed by increasing coverage by providing explicit >> tests. >> >> Coming back to the releases: >> I have no objections, if you do a "release" locally and automate the >> validation on the CI server (Which effectively would be your proposal to do >> the first 12 steps on local hardware and the 13th on the CI server). I even >> think that's a good idea ... There could be one step for building a release >> from a given "git tag" for every build system and generic means to compare >> tar.gz and zips produced by any build system with that of another (ideally >> with better output than just a plain "true/false"). This would even help to >> iron out the last potentially existing bumps out of the Maven distribution. >> >> Chris >> >> >> >> Am 02.04.20, 07:59 schrieb "Alex Harui" <[email protected]>: >> >> Hi, >> >> This is my attempt to explain what goes into a release in hopes that >> we can understand and agree on what our release process is. It became >> apparent in my reading of the wiki page with the new Maven steps and in >> talking with Harbs today that there are still many misunderstandings about >> what we do to create a release. I don't generally like writing >> instructions in English because it is easy to be ambiguous. All of the >> steps that we use to create releases had been captured in Ant scripts in a >> much more explicit way, IMO, but I took the time to write them down in >> English here: >> https://github.com/apache/royale-asjs/wiki/Task-List-For-Royale-Releases >> >> I did this quickly by scanning the CI steps, the new Maven steps and >> the Ant scripts used in prior releases so there could certainly be mistakes >> and missed steps. If I did my math right, the RM for 0.9.7 will have to >> complete over 100 tasks (essentially, typing a command-line 100 times). >> Future RMs, when we don't have to release build-tools, will have about 92 >> steps. And I did not include voter verification checks the RM should run >> before opening a vote (verifying that the artifacts download and match >> their checksums, etc). As an RM, I run a bunch of tests on the RC before >> sending out the vote. Maybe we should add those to the task list. >> >> I think there has been confusion about the use of Ant in the release >> process. Because I was the RM for the first set of FlexJS/Royale releases, >> and I'm a lazy person who hates typing at the command line, I created Ant >> scripts to execute these 100 steps. But I agree that it is not a >> requirement that other RMs must use the Ant scripts for these commands. If >> you are the RM and like typing, go ahead. >> >> Then we found out that other people couldn't get through this task >> list. I think the 3 people who tried were having trouble with Maven >> uploads and downloads. So what I did was put the first 40 steps or so into >> Jenkins jobs. And by doing that, Piotr was able to produce our last >> release. And that also saves on manually typing commands. But again, >> going forward, the RM gets to choose how they want to execute these steps. >> >> If you scan the set of steps, you'll see that "ant" is only in there >> once. I believe the recent threads have been about this single command out >> of the 100+ commands. This is why this has been so frustrating to me. I >> believe there is a solid technical reason for that one command: it proves >> that the build.xml files in the source packages can build the .tar.gz that >> are useful to NPM and IDE users who use Ant and want to test a change. >> >> I think of it as code-coverage. If we had code-coverage tools, we >> would ask that the RM complete as much of the automated code-coverage >> testing as possible before posting the release for a vote. That one >> command increases our code coverage by running the build.xml files. We >> should be always working to increase automated code coverage in the RC. >> Certainly for me as RM, I will gladly watch TV as the automated tests run >> because a failed RC means going back through many of the first 25 commands >> again and wastes other people's time. Each RC is more emails to read and >> more time from the voters and testers. >> >> If there are other ways for the RM to get the same or better code >> coverage on the build.xml files before posting the RC, we can discuss those >> options. >> >> I am hopeful we can all agree on these simple principles: Strive for >> better code coverage and fewer failed RCs. Royale's main purpose is to >> save other people time. Let's do that in creating releases too. >> >> One issue that was brought up recently was whether it is a good >> decision to have the RM test all of the build platforms we support. >> Suppose we add some other build system support or more in the future? >> Again, the code coverage principle applies here, but also, I would like us >> to retain feature parity, and I also hope for as few RCs and votes as >> possible. So instead of having separate votes/features/release-dates for >> the Maven artifacts vs the Ant artifacts vs the SomeFutureBuildTech >> artifacts, I think we should have one vote and keep them all in sync. If >> we do ever get around to monthly or bi-monthly releases, I think separate >> build platform releases would be too much work. >> >> But consider this thought I just had today: the RM doesn't really >> have to choose to do all 100 commands on a local machine or with Ant >> scripts or do the first 40 via CI. The RM can actually pick and choose >> commands to run on the CI server. The CI Jenkins jobs are not a >> separate/alternative release process, they are just another way of >> executing the first 40 steps. Using CI jobs actually requires additional >> command-line cut-and-paste to push commits on the CI server and to sign and >> validate binaries locally, but that's the trade-off of not having to >> configure your machine to successfully run all of the automated tests and >> build systems, and being able to run a command by filling in the version >> number and rc number and hitting the "ok" button instead of making sure you >> got the whole command typed in correctly. >> >> So, an RM can run the first 25 steps locally, then go the CI server >> and run what is now Jenkins Job "Royale_Release_Step_013" (no need to run >> the first 12) and it will run tasks 26 through 32, and if it is successful, >> then the RM has proven code coverage of the build.xml files. (If the >> resulting tar.gz and zips are not posted, then the RM should verify that >> they match the ones from Maven distribution). I would encourage RMs to >> also use the CI jobs that generate the emails to make sure the subject and >> content is correct and contains the usual instructions so we have >> consistency. Maybe someday there will be CI jobs to do the last 60+ steps >> if that helps. We could add a Jenkins job that runs an Ant build on RC >> artifacts on dist.a.o as well. >> >> I would like you all to help maintain the list of 100 steps and other >> documents related to the release process, and improve the CI jobs and Ant >> steps if it helps you be a more efficient RM. I am hopeful that now that I >> have hopefully explained our release process better, that we can see that >> these 100+ steps just have to be done in some way. The RM can figure out >> what way works best for them, but they must get through all of them. >> >> Thanks, >> -Alex >> >> >> >> >> >> >> >> >> > > -- > Carlos Rovira > http://about.me/carlosrovira
