Hi Harbs, I think what we're trying to say is that until now we released with Maven and Ant, and that was hiding a flaw in Maven (SVG example). So that means what we were trying to cover was not covered clearly, so the premise is not right.
El jue., 2 abr. 2020 a las 14:56, Harbs (<[email protected]>) escribió: > No one is arguing that we shouldn’t add more tests. > > Please let’s not make it seem like there’s a disagreement about that. > > > On Apr 2, 2020, at 10:46 AM, Carlos Rovira <[email protected]> > wrote: > > > > Hi Alex, > > > > first, many thanks for the detailed email. I'll comment on this later as > I > > have more time. > > > > For now, to add up a recent example on what Chris commented: If you all > > remember a week ago I was trying to use SVG Images in a blog example that > > was published 2 days ago. Nobody tried SVG Images before building with > > maven, I know that since maven was not properly configured and using that > > component from Maven was failing with an RTE. Probably we have more > things > > not working the same way when build from Maven and Ant, and that's > > something that will need people using that code paths in test > applications > > (or in their own apps) to see if things works properly. > > > > I was recently introduced to "examples-integrationtest" by Chris, that I > > plan to use soon as I can. I think is a great idea, since you get a > Firefox > > running test interface of the real use of some concrete royale code. I > > think passed until now unnoticed by all of us, and seems a powerful tool. > > There's already an example about FlexStore with some basic assertions. > > > > Again, thanks, and will comment on the rest later > > > > Carlos > > > > > > > > > > El jue., 2 abr. 2020 a las 9:20, Christofer Dutz (< > [email protected]>) > > escribió: > > > >> Hi Alex, > >> > >> just a point you are bringing up: "Code coverage". > >> > >> I strongly dislike the idea of "asjs" effectively being the test for the > >> compiler. The reasoning behind this is: yes you do get more code > covered, > >> but only the happy-path (ideally) and even if things go wrong, the end > >> results aren't tested. Did add a module to asjs years ago > >> ("examples-integrationtest") that deployed the examples in a tomcat > server > >> then opens a Firefox browser and clicks through 2 of the examples (I > added > >> two dummy tests as an example, but seems no one touched this after me). > I > >> did this because I remember us working on asjs for weeks without anyone > >> noticing the compiler wasn't producing runnable code ... same with the > >> little unit-tests that are still run for every example, that simply > check > >> if an output is generated, because we had a prolonged period of time > where > >> we were all working on different parts, but for quite some time the > >> application compilation just didn't output anything and no one noticed. > >> > >> So coverage is nothing without assertions (my opinion) ... ok ... it's > >> slightly better than no coverage, but not much, in my opinion. > >> > >> I think in parallel to this release discussions I have seen numerous > >> threads about someone doing something that broke something for someone > >> else. This could be addressed by increasing coverage by providing > explicit > >> tests. > >> > >> Coming back to the releases: > >> I have no objections, if you do a "release" locally and automate the > >> validation on the CI server (Which effectively would be your proposal > to do > >> the first 12 steps on local hardware and the 13th on the CI server). I > even > >> think that's a good idea ... There could be one step for building a > release > >> from a given "git tag" for every build system and generic means to > compare > >> tar.gz and zips produced by any build system with that of another > (ideally > >> with better output than just a plain "true/false"). This would even > help to > >> iron out the last potentially existing bumps out of the Maven > distribution. > >> > >> Chris > >> > >> > >> > >> Am 02.04.20, 07:59 schrieb "Alex Harui" <[email protected]>: > >> > >> Hi, > >> > >> This is my attempt to explain what goes into a release in hopes that > >> we can understand and agree on what our release process is. It became > >> apparent in my reading of the wiki page with the new Maven steps and in > >> talking with Harbs today that there are still many misunderstandings > about > >> what we do to create a release. I don't generally like writing > >> instructions in English because it is easy to be ambiguous. All of the > >> steps that we use to create releases had been captured in Ant scripts > in a > >> much more explicit way, IMO, but I took the time to write them down in > >> English here: > >> > https://github.com/apache/royale-asjs/wiki/Task-List-For-Royale-Releases > >> > >> I did this quickly by scanning the CI steps, the new Maven steps and > >> the Ant scripts used in prior releases so there could certainly be > mistakes > >> and missed steps. If I did my math right, the RM for 0.9.7 will have to > >> complete over 100 tasks (essentially, typing a command-line 100 times). > >> Future RMs, when we don't have to release build-tools, will have about > 92 > >> steps. And I did not include voter verification checks the RM should > run > >> before opening a vote (verifying that the artifacts download and match > >> their checksums, etc). As an RM, I run a bunch of tests on the RC > before > >> sending out the vote. Maybe we should add those to the task list. > >> > >> I think there has been confusion about the use of Ant in the release > >> process. Because I was the RM for the first set of FlexJS/Royale > releases, > >> and I'm a lazy person who hates typing at the command line, I created > Ant > >> scripts to execute these 100 steps. But I agree that it is not a > >> requirement that other RMs must use the Ant scripts for these > commands. If > >> you are the RM and like typing, go ahead. > >> > >> Then we found out that other people couldn't get through this task > >> list. I think the 3 people who tried were having trouble with Maven > >> uploads and downloads. So what I did was put the first 40 steps or so > into > >> Jenkins jobs. And by doing that, Piotr was able to produce our last > >> release. And that also saves on manually typing commands. But again, > >> going forward, the RM gets to choose how they want to execute these > steps. > >> > >> If you scan the set of steps, you'll see that "ant" is only in there > >> once. I believe the recent threads have been about this single command > out > >> of the 100+ commands. This is why this has been so frustrating to me. > I > >> believe there is a solid technical reason for that one command: it > proves > >> that the build.xml files in the source packages can build the .tar.gz > that > >> are useful to NPM and IDE users who use Ant and want to test a change. > >> > >> I think of it as code-coverage. If we had code-coverage tools, we > >> would ask that the RM complete as much of the automated code-coverage > >> testing as possible before posting the release for a vote. That one > >> command increases our code coverage by running the build.xml files. We > >> should be always working to increase automated code coverage in the RC. > >> Certainly for me as RM, I will gladly watch TV as the automated tests > run > >> because a failed RC means going back through many of the first 25 > commands > >> again and wastes other people's time. Each RC is more emails to read > and > >> more time from the voters and testers. > >> > >> If there are other ways for the RM to get the same or better code > >> coverage on the build.xml files before posting the RC, we can discuss > those > >> options. > >> > >> I am hopeful we can all agree on these simple principles: Strive for > >> better code coverage and fewer failed RCs. Royale's main purpose is to > >> save other people time. Let's do that in creating releases too. > >> > >> One issue that was brought up recently was whether it is a good > >> decision to have the RM test all of the build platforms we support. > >> Suppose we add some other build system support or more in the future? > >> Again, the code coverage principle applies here, but also, I would like > us > >> to retain feature parity, and I also hope for as few RCs and votes as > >> possible. So instead of having separate votes/features/release-dates > for > >> the Maven artifacts vs the Ant artifacts vs the SomeFutureBuildTech > >> artifacts, I think we should have one vote and keep them all in sync. > If > >> we do ever get around to monthly or bi-monthly releases, I think > separate > >> build platform releases would be too much work. > >> > >> But consider this thought I just had today: the RM doesn't really > >> have to choose to do all 100 commands on a local machine or with Ant > >> scripts or do the first 40 via CI. The RM can actually pick and choose > >> commands to run on the CI server. The CI Jenkins jobs are not a > >> separate/alternative release process, they are just another way of > >> executing the first 40 steps. Using CI jobs actually requires > additional > >> command-line cut-and-paste to push commits on the CI server and to sign > and > >> validate binaries locally, but that's the trade-off of not having to > >> configure your machine to successfully run all of the automated tests > and > >> build systems, and being able to run a command by filling in the version > >> number and rc number and hitting the "ok" button instead of making sure > you > >> got the whole command typed in correctly. > >> > >> So, an RM can run the first 25 steps locally, then go the CI server > >> and run what is now Jenkins Job "Royale_Release_Step_013" (no need to > run > >> the first 12) and it will run tasks 26 through 32, and if it is > successful, > >> then the RM has proven code coverage of the build.xml files. (If the > >> resulting tar.gz and zips are not posted, then the RM should verify that > >> they match the ones from Maven distribution). I would encourage RMs to > >> also use the CI jobs that generate the emails to make sure the subject > and > >> content is correct and contains the usual instructions so we have > >> consistency. Maybe someday there will be CI jobs to do the last 60+ > steps > >> if that helps. We could add a Jenkins job that runs an Ant build on RC > >> artifacts on dist.a.o as well. > >> > >> I would like you all to help maintain the list of 100 steps and other > >> documents related to the release process, and improve the CI jobs and > Ant > >> steps if it helps you be a more efficient RM. I am hopeful that now > that I > >> have hopefully explained our release process better, that we can see > that > >> these 100+ steps just have to be done in some way. The RM can figure > out > >> what way works best for them, but they must get through all of them. > >> > >> Thanks, > >> -Alex > >> > >> > >> > >> > >> > >> > >> > >> > >> > > > > -- > > Carlos Rovira > > http://about.me/carlosrovira > > -- Carlos Rovira http://about.me/carlosrovira
