We’re arguing semantics. This has nothing to do with Alex’s main points and this discussion is distracting from the main issues.
> On Apr 2, 2020, at 4:25 PM, Carlos Rovira <[email protected]> wrote: > > Harbs, > > again, don't think we're talking about removing code-coverage from anything. > Code coverage must be in all Maven and Ant. Equally. > What we're trying to say is that what is actually called "code coverage" is > not really code coverage. > > > El jue., 2 abr. 2020 a las 15:09, Harbs (<[email protected]>) escribió: > >> Adding more coverage for Maven is good. >> >> Removing coverage for Ant is not. >> >> Do you agree? >> >>> On Apr 2, 2020, at 4:07 PM, Carlos Rovira <[email protected]> >> wrote: >>> >>> Hi Harbs, >>> >>> I think what we're trying to say is that until now we released with Maven >>> and Ant, and that was hiding a flaw in Maven (SVG example). So that means >>> what we were trying to cover was not covered clearly, so the premise is >> not >>> right. >>> >>> >>> >>> El jue., 2 abr. 2020 a las 14:56, Harbs (<[email protected]>) >> escribió: >>> >>>> No one is arguing that we shouldn’t add more tests. >>>> >>>> Please let’s not make it seem like there’s a disagreement about that. >>>> >>>>> On Apr 2, 2020, at 10:46 AM, Carlos Rovira <[email protected]> >>>> wrote: >>>>> >>>>> Hi Alex, >>>>> >>>>> first, many thanks for the detailed email. I'll comment on this later >> as >>>> I >>>>> have more time. >>>>> >>>>> For now, to add up a recent example on what Chris commented: If you all >>>>> remember a week ago I was trying to use SVG Images in a blog example >> that >>>>> was published 2 days ago. Nobody tried SVG Images before building with >>>>> maven, I know that since maven was not properly configured and using >> that >>>>> component from Maven was failing with an RTE. Probably we have more >>>> things >>>>> not working the same way when build from Maven and Ant, and that's >>>>> something that will need people using that code paths in test >>>> applications >>>>> (or in their own apps) to see if things works properly. >>>>> >>>>> I was recently introduced to "examples-integrationtest" by Chris, that >> I >>>>> plan to use soon as I can. I think is a great idea, since you get a >>>> Firefox >>>>> running test interface of the real use of some concrete royale code. I >>>>> think passed until now unnoticed by all of us, and seems a powerful >> tool. >>>>> There's already an example about FlexStore with some basic assertions. >>>>> >>>>> Again, thanks, and will comment on the rest later >>>>> >>>>> Carlos >>>>> >>>>> >>>>> >>>>> >>>>> El jue., 2 abr. 2020 a las 9:20, Christofer Dutz (< >>>> [email protected]>) >>>>> escribió: >>>>> >>>>>> Hi Alex, >>>>>> >>>>>> just a point you are bringing up: "Code coverage". >>>>>> >>>>>> I strongly dislike the idea of "asjs" effectively being the test for >> the >>>>>> compiler. The reasoning behind this is: yes you do get more code >>>> covered, >>>>>> but only the happy-path (ideally) and even if things go wrong, the end >>>>>> results aren't tested. Did add a module to asjs years ago >>>>>> ("examples-integrationtest") that deployed the examples in a tomcat >>>> server >>>>>> then opens a Firefox browser and clicks through 2 of the examples (I >>>> added >>>>>> two dummy tests as an example, but seems no one touched this after >> me). >>>> I >>>>>> did this because I remember us working on asjs for weeks without >> anyone >>>>>> noticing the compiler wasn't producing runnable code ... same with the >>>>>> little unit-tests that are still run for every example, that simply >>>> check >>>>>> if an output is generated, because we had a prolonged period of time >>>> where >>>>>> we were all working on different parts, but for quite some time the >>>>>> application compilation just didn't output anything and no one >> noticed. >>>>>> >>>>>> So coverage is nothing without assertions (my opinion) ... ok ... it's >>>>>> slightly better than no coverage, but not much, in my opinion. >>>>>> >>>>>> I think in parallel to this release discussions I have seen numerous >>>>>> threads about someone doing something that broke something for someone >>>>>> else. This could be addressed by increasing coverage by providing >>>> explicit >>>>>> tests. >>>>>> >>>>>> Coming back to the releases: >>>>>> I have no objections, if you do a "release" locally and automate the >>>>>> validation on the CI server (Which effectively would be your proposal >>>> to do >>>>>> the first 12 steps on local hardware and the 13th on the CI server). I >>>> even >>>>>> think that's a good idea ... There could be one step for building a >>>> release >>>>>> from a given "git tag" for every build system and generic means to >>>> compare >>>>>> tar.gz and zips produced by any build system with that of another >>>> (ideally >>>>>> with better output than just a plain "true/false"). This would even >>>> help to >>>>>> iron out the last potentially existing bumps out of the Maven >>>> distribution. >>>>>> >>>>>> Chris >>>>>> >>>>>> >>>>>> >>>>>> Am 02.04.20, 07:59 schrieb "Alex Harui" <[email protected]>: >>>>>> >>>>>> Hi, >>>>>> >>>>>> This is my attempt to explain what goes into a release in hopes that >>>>>> we can understand and agree on what our release process is. It became >>>>>> apparent in my reading of the wiki page with the new Maven steps and >> in >>>>>> talking with Harbs today that there are still many misunderstandings >>>> about >>>>>> what we do to create a release. I don't generally like writing >>>>>> instructions in English because it is easy to be ambiguous. All of >> the >>>>>> steps that we use to create releases had been captured in Ant scripts >>>> in a >>>>>> much more explicit way, IMO, but I took the time to write them down in >>>>>> English here: >>>>>> >>>> >> https://github.com/apache/royale-asjs/wiki/Task-List-For-Royale-Releases >>>>>> >>>>>> I did this quickly by scanning the CI steps, the new Maven steps and >>>>>> the Ant scripts used in prior releases so there could certainly be >>>> mistakes >>>>>> and missed steps. If I did my math right, the RM for 0.9.7 will have >> to >>>>>> complete over 100 tasks (essentially, typing a command-line 100 >> times). >>>>>> Future RMs, when we don't have to release build-tools, will have about >>>> 92 >>>>>> steps. And I did not include voter verification checks the RM should >>>> run >>>>>> before opening a vote (verifying that the artifacts download and match >>>>>> their checksums, etc). As an RM, I run a bunch of tests on the RC >>>> before >>>>>> sending out the vote. Maybe we should add those to the task list. >>>>>> >>>>>> I think there has been confusion about the use of Ant in the release >>>>>> process. Because I was the RM for the first set of FlexJS/Royale >>>> releases, >>>>>> and I'm a lazy person who hates typing at the command line, I created >>>> Ant >>>>>> scripts to execute these 100 steps. But I agree that it is not a >>>>>> requirement that other RMs must use the Ant scripts for these >>>> commands. If >>>>>> you are the RM and like typing, go ahead. >>>>>> >>>>>> Then we found out that other people couldn't get through this task >>>>>> list. I think the 3 people who tried were having trouble with Maven >>>>>> uploads and downloads. So what I did was put the first 40 steps or so >>>> into >>>>>> Jenkins jobs. And by doing that, Piotr was able to produce our last >>>>>> release. And that also saves on manually typing commands. But again, >>>>>> going forward, the RM gets to choose how they want to execute these >>>> steps. >>>>>> >>>>>> If you scan the set of steps, you'll see that "ant" is only in there >>>>>> once. I believe the recent threads have been about this single >> command >>>> out >>>>>> of the 100+ commands. This is why this has been so frustrating to me. >>>> I >>>>>> believe there is a solid technical reason for that one command: it >>>> proves >>>>>> that the build.xml files in the source packages can build the .tar.gz >>>> that >>>>>> are useful to NPM and IDE users who use Ant and want to test a change. >>>>>> >>>>>> I think of it as code-coverage. If we had code-coverage tools, we >>>>>> would ask that the RM complete as much of the automated code-coverage >>>>>> testing as possible before posting the release for a vote. That one >>>>>> command increases our code coverage by running the build.xml files. >> We >>>>>> should be always working to increase automated code coverage in the >> RC. >>>>>> Certainly for me as RM, I will gladly watch TV as the automated tests >>>> run >>>>>> because a failed RC means going back through many of the first 25 >>>> commands >>>>>> again and wastes other people's time. Each RC is more emails to read >>>> and >>>>>> more time from the voters and testers. >>>>>> >>>>>> If there are other ways for the RM to get the same or better code >>>>>> coverage on the build.xml files before posting the RC, we can discuss >>>> those >>>>>> options. >>>>>> >>>>>> I am hopeful we can all agree on these simple principles: Strive >> for >>>>>> better code coverage and fewer failed RCs. Royale's main purpose is >> to >>>>>> save other people time. Let's do that in creating releases too. >>>>>> >>>>>> One issue that was brought up recently was whether it is a good >>>>>> decision to have the RM test all of the build platforms we support. >>>>>> Suppose we add some other build system support or more in the future? >>>>>> Again, the code coverage principle applies here, but also, I would >> like >>>> us >>>>>> to retain feature parity, and I also hope for as few RCs and votes as >>>>>> possible. So instead of having separate votes/features/release-dates >>>> for >>>>>> the Maven artifacts vs the Ant artifacts vs the SomeFutureBuildTech >>>>>> artifacts, I think we should have one vote and keep them all in sync. >>>> If >>>>>> we do ever get around to monthly or bi-monthly releases, I think >>>> separate >>>>>> build platform releases would be too much work. >>>>>> >>>>>> But consider this thought I just had today: the RM doesn't really >>>>>> have to choose to do all 100 commands on a local machine or with Ant >>>>>> scripts or do the first 40 via CI. The RM can actually pick and >> choose >>>>>> commands to run on the CI server. The CI Jenkins jobs are not a >>>>>> separate/alternative release process, they are just another way of >>>>>> executing the first 40 steps. Using CI jobs actually requires >>>> additional >>>>>> command-line cut-and-paste to push commits on the CI server and to >> sign >>>> and >>>>>> validate binaries locally, but that's the trade-off of not having to >>>>>> configure your machine to successfully run all of the automated tests >>>> and >>>>>> build systems, and being able to run a command by filling in the >> version >>>>>> number and rc number and hitting the "ok" button instead of making >> sure >>>> you >>>>>> got the whole command typed in correctly. >>>>>> >>>>>> So, an RM can run the first 25 steps locally, then go the CI server >>>>>> and run what is now Jenkins Job "Royale_Release_Step_013" (no need to >>>> run >>>>>> the first 12) and it will run tasks 26 through 32, and if it is >>>> successful, >>>>>> then the RM has proven code coverage of the build.xml files. (If the >>>>>> resulting tar.gz and zips are not posted, then the RM should verify >> that >>>>>> they match the ones from Maven distribution). I would encourage RMs >> to >>>>>> also use the CI jobs that generate the emails to make sure the subject >>>> and >>>>>> content is correct and contains the usual instructions so we have >>>>>> consistency. Maybe someday there will be CI jobs to do the last 60+ >>>> steps >>>>>> if that helps. We could add a Jenkins job that runs an Ant build on >> RC >>>>>> artifacts on dist.a.o as well. >>>>>> >>>>>> I would like you all to help maintain the list of 100 steps and >> other >>>>>> documents related to the release process, and improve the CI jobs and >>>> Ant >>>>>> steps if it helps you be a more efficient RM. I am hopeful that now >>>> that I >>>>>> have hopefully explained our release process better, that we can see >>>> that >>>>>> these 100+ steps just have to be done in some way. The RM can figure >>>> out >>>>>> what way works best for them, but they must get through all of them. >>>>>> >>>>>> Thanks, >>>>>> -Alex >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Carlos Rovira >>>>> http://about.me/carlosrovira >>>> >>>> >>> >>> -- >>> Carlos Rovira >>> http://about.me/carlosrovira >> >> > > -- > Carlos Rovira > http://about.me/carlosrovira
