Adding more coverage for Maven is good. Removing coverage for Ant is not.
Do you agree? > On Apr 2, 2020, at 4:07 PM, Carlos Rovira <[email protected]> wrote: > > Hi Harbs, > > I think what we're trying to say is that until now we released with Maven > and Ant, and that was hiding a flaw in Maven (SVG example). So that means > what we were trying to cover was not covered clearly, so the premise is not > right. > > > > El jue., 2 abr. 2020 a las 14:56, Harbs (<[email protected]>) escribió: > >> No one is arguing that we shouldn’t add more tests. >> >> Please let’s not make it seem like there’s a disagreement about that. >> >>> On Apr 2, 2020, at 10:46 AM, Carlos Rovira <[email protected]> >> wrote: >>> >>> Hi Alex, >>> >>> first, many thanks for the detailed email. I'll comment on this later as >> I >>> have more time. >>> >>> For now, to add up a recent example on what Chris commented: If you all >>> remember a week ago I was trying to use SVG Images in a blog example that >>> was published 2 days ago. Nobody tried SVG Images before building with >>> maven, I know that since maven was not properly configured and using that >>> component from Maven was failing with an RTE. Probably we have more >> things >>> not working the same way when build from Maven and Ant, and that's >>> something that will need people using that code paths in test >> applications >>> (or in their own apps) to see if things works properly. >>> >>> I was recently introduced to "examples-integrationtest" by Chris, that I >>> plan to use soon as I can. I think is a great idea, since you get a >> Firefox >>> running test interface of the real use of some concrete royale code. I >>> think passed until now unnoticed by all of us, and seems a powerful tool. >>> There's already an example about FlexStore with some basic assertions. >>> >>> Again, thanks, and will comment on the rest later >>> >>> Carlos >>> >>> >>> >>> >>> El jue., 2 abr. 2020 a las 9:20, Christofer Dutz (< >> [email protected]>) >>> escribió: >>> >>>> Hi Alex, >>>> >>>> just a point you are bringing up: "Code coverage". >>>> >>>> I strongly dislike the idea of "asjs" effectively being the test for the >>>> compiler. The reasoning behind this is: yes you do get more code >> covered, >>>> but only the happy-path (ideally) and even if things go wrong, the end >>>> results aren't tested. Did add a module to asjs years ago >>>> ("examples-integrationtest") that deployed the examples in a tomcat >> server >>>> then opens a Firefox browser and clicks through 2 of the examples (I >> added >>>> two dummy tests as an example, but seems no one touched this after me). >> I >>>> did this because I remember us working on asjs for weeks without anyone >>>> noticing the compiler wasn't producing runnable code ... same with the >>>> little unit-tests that are still run for every example, that simply >> check >>>> if an output is generated, because we had a prolonged period of time >> where >>>> we were all working on different parts, but for quite some time the >>>> application compilation just didn't output anything and no one noticed. >>>> >>>> So coverage is nothing without assertions (my opinion) ... ok ... it's >>>> slightly better than no coverage, but not much, in my opinion. >>>> >>>> I think in parallel to this release discussions I have seen numerous >>>> threads about someone doing something that broke something for someone >>>> else. This could be addressed by increasing coverage by providing >> explicit >>>> tests. >>>> >>>> Coming back to the releases: >>>> I have no objections, if you do a "release" locally and automate the >>>> validation on the CI server (Which effectively would be your proposal >> to do >>>> the first 12 steps on local hardware and the 13th on the CI server). I >> even >>>> think that's a good idea ... There could be one step for building a >> release >>>> from a given "git tag" for every build system and generic means to >> compare >>>> tar.gz and zips produced by any build system with that of another >> (ideally >>>> with better output than just a plain "true/false"). This would even >> help to >>>> iron out the last potentially existing bumps out of the Maven >> distribution. >>>> >>>> Chris >>>> >>>> >>>> >>>> Am 02.04.20, 07:59 schrieb "Alex Harui" <[email protected]>: >>>> >>>> Hi, >>>> >>>> This is my attempt to explain what goes into a release in hopes that >>>> we can understand and agree on what our release process is. It became >>>> apparent in my reading of the wiki page with the new Maven steps and in >>>> talking with Harbs today that there are still many misunderstandings >> about >>>> what we do to create a release. I don't generally like writing >>>> instructions in English because it is easy to be ambiguous. All of the >>>> steps that we use to create releases had been captured in Ant scripts >> in a >>>> much more explicit way, IMO, but I took the time to write them down in >>>> English here: >>>> >> https://github.com/apache/royale-asjs/wiki/Task-List-For-Royale-Releases >>>> >>>> I did this quickly by scanning the CI steps, the new Maven steps and >>>> the Ant scripts used in prior releases so there could certainly be >> mistakes >>>> and missed steps. If I did my math right, the RM for 0.9.7 will have to >>>> complete over 100 tasks (essentially, typing a command-line 100 times). >>>> Future RMs, when we don't have to release build-tools, will have about >> 92 >>>> steps. And I did not include voter verification checks the RM should >> run >>>> before opening a vote (verifying that the artifacts download and match >>>> their checksums, etc). As an RM, I run a bunch of tests on the RC >> before >>>> sending out the vote. Maybe we should add those to the task list. >>>> >>>> I think there has been confusion about the use of Ant in the release >>>> process. Because I was the RM for the first set of FlexJS/Royale >> releases, >>>> and I'm a lazy person who hates typing at the command line, I created >> Ant >>>> scripts to execute these 100 steps. But I agree that it is not a >>>> requirement that other RMs must use the Ant scripts for these >> commands. If >>>> you are the RM and like typing, go ahead. >>>> >>>> Then we found out that other people couldn't get through this task >>>> list. I think the 3 people who tried were having trouble with Maven >>>> uploads and downloads. So what I did was put the first 40 steps or so >> into >>>> Jenkins jobs. And by doing that, Piotr was able to produce our last >>>> release. And that also saves on manually typing commands. But again, >>>> going forward, the RM gets to choose how they want to execute these >> steps. >>>> >>>> If you scan the set of steps, you'll see that "ant" is only in there >>>> once. I believe the recent threads have been about this single command >> out >>>> of the 100+ commands. This is why this has been so frustrating to me. >> I >>>> believe there is a solid technical reason for that one command: it >> proves >>>> that the build.xml files in the source packages can build the .tar.gz >> that >>>> are useful to NPM and IDE users who use Ant and want to test a change. >>>> >>>> I think of it as code-coverage. If we had code-coverage tools, we >>>> would ask that the RM complete as much of the automated code-coverage >>>> testing as possible before posting the release for a vote. That one >>>> command increases our code coverage by running the build.xml files. We >>>> should be always working to increase automated code coverage in the RC. >>>> Certainly for me as RM, I will gladly watch TV as the automated tests >> run >>>> because a failed RC means going back through many of the first 25 >> commands >>>> again and wastes other people's time. Each RC is more emails to read >> and >>>> more time from the voters and testers. >>>> >>>> If there are other ways for the RM to get the same or better code >>>> coverage on the build.xml files before posting the RC, we can discuss >> those >>>> options. >>>> >>>> I am hopeful we can all agree on these simple principles: Strive for >>>> better code coverage and fewer failed RCs. Royale's main purpose is to >>>> save other people time. Let's do that in creating releases too. >>>> >>>> One issue that was brought up recently was whether it is a good >>>> decision to have the RM test all of the build platforms we support. >>>> Suppose we add some other build system support or more in the future? >>>> Again, the code coverage principle applies here, but also, I would like >> us >>>> to retain feature parity, and I also hope for as few RCs and votes as >>>> possible. So instead of having separate votes/features/release-dates >> for >>>> the Maven artifacts vs the Ant artifacts vs the SomeFutureBuildTech >>>> artifacts, I think we should have one vote and keep them all in sync. >> If >>>> we do ever get around to monthly or bi-monthly releases, I think >> separate >>>> build platform releases would be too much work. >>>> >>>> But consider this thought I just had today: the RM doesn't really >>>> have to choose to do all 100 commands on a local machine or with Ant >>>> scripts or do the first 40 via CI. The RM can actually pick and choose >>>> commands to run on the CI server. The CI Jenkins jobs are not a >>>> separate/alternative release process, they are just another way of >>>> executing the first 40 steps. Using CI jobs actually requires >> additional >>>> command-line cut-and-paste to push commits on the CI server and to sign >> and >>>> validate binaries locally, but that's the trade-off of not having to >>>> configure your machine to successfully run all of the automated tests >> and >>>> build systems, and being able to run a command by filling in the version >>>> number and rc number and hitting the "ok" button instead of making sure >> you >>>> got the whole command typed in correctly. >>>> >>>> So, an RM can run the first 25 steps locally, then go the CI server >>>> and run what is now Jenkins Job "Royale_Release_Step_013" (no need to >> run >>>> the first 12) and it will run tasks 26 through 32, and if it is >> successful, >>>> then the RM has proven code coverage of the build.xml files. (If the >>>> resulting tar.gz and zips are not posted, then the RM should verify that >>>> they match the ones from Maven distribution). I would encourage RMs to >>>> also use the CI jobs that generate the emails to make sure the subject >> and >>>> content is correct and contains the usual instructions so we have >>>> consistency. Maybe someday there will be CI jobs to do the last 60+ >> steps >>>> if that helps. We could add a Jenkins job that runs an Ant build on RC >>>> artifacts on dist.a.o as well. >>>> >>>> I would like you all to help maintain the list of 100 steps and other >>>> documents related to the release process, and improve the CI jobs and >> Ant >>>> steps if it helps you be a more efficient RM. I am hopeful that now >> that I >>>> have hopefully explained our release process better, that we can see >> that >>>> these 100+ steps just have to be done in some way. The RM can figure >> out >>>> what way works best for them, but they must get through all of them. >>>> >>>> Thanks, >>>> -Alex >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> Carlos Rovira >>> http://about.me/carlosrovira >> >> > > -- > Carlos Rovira > http://about.me/carlosrovira
