Why don’t we hold off on discussing this until Yishay and I have both gone through a release process?
I’m sure the necessity (or lack thereof) will become clear to us once we’ve done it. > On Apr 3, 2020, at 11:23 AM, Carlos Rovira <[email protected]> wrote: > > Hi Alex, > > trying to summarize again with this plan seems to me unnecessary, in a way > that maybe until now nobody exposed and could be the key to understand the > problem: > > if we ask ourselves: "Are we generating the same artifacts between build > systems at bit level?", the response is "No". They should be mostly the > same for an user to work with it, but after working on a release it was > clear to me that each build system has its own "idiosyncrasy" (to say it > some way), since we us to have files like metainfs, xmls, or other things > that are not generated the same way. > > "are we creating reproducible builds?". "No". Since the compiler is not > prepared for it. Due to differences in OS (and possibly JDK Vendor) the > order generated in some files is different (as already exposed many days > ago). So reproducible builds in 0.9.6 are not real. "what needs to be made > to have reproducible builds?". "Fix the compiler is needed to implement > it". > > Far beyond, "If we fix the compiler to have real reproducible builds will > we have reproducible builds between build systems?". "No, will be only > reproducible from a concrete build system perspective". So builds for Ant > are reproducible for Ant, and builds done with Maven are reproducible with > Maven, but are not comparable between them due to latest point, each build > system generate slightly different artifacts. Checking jars for binary > equality will fail (just opening with a tool like intellij will show you > the problem easily). > > (subsection: since reproducibility seems more work we can left for post > 1.0.0) > > So "if you make a release using one build system have sense to mix with > other build system during the release?", "No, since as we are creating > different artifacts, just running a full build, when release is done, to > test the rest of build systems are working is enough to check it". Or "it > no has sense to do release steps for all build systems since we just need > to push one set of artifacts and we are really not checking any relation > between them since are different things at byte level although can work the > same for an user" > > If the problem is to save time to the rest of people verifying the release, > the RM can do a full build with *all* build systems to check the integrity > and that build systems are working as we all expect, so RCs are in the same > state as running the 100 steps. That will far easy and quick than doing 100 > commands that really are not performing what we expect from what I > explained here. > > In resume, both ways are the same, since reproducible builds are only > affecting to the build system from what you're building and the so-called > "reproducible artifacts" and not "comparable" with same artifacts generated > by other build systems, so artifacts done with different systems are > essentially (bit level) not equal). > > If after all the evidences exposed here, you still insists in impose a 100 > command recipe, I honestly will not understand. > > In exchange, if you (or other) feels better doing 100 commands when acting > as RM, is completely up to you, while don't try to impose to the rest of > the project. > > HTH > > Thanks > > Carlos > > > > > > El vie., 3 abr. 2020 a las 6:35, Alex Harui (<[email protected]>) > escribió: > >> I honestly can't think of any steps we can do without. If you can explain >> technically why some step isn't needed, then we can discuss it. I'm sorry >> you don't agree. I hope the rest of the PMC will agree that we have to do >> all of these steps and support the RM in doing so. >> >> One way to not have to actually type 90+ command is to use the Ant steps >> or CI jobs. Ideas on how to do less typing to execute these 90+ commands >> are welcome. >> >> -Alex >> >> On 4/2/20, 4:22 PM, "Carlos Rovira" <[email protected]> wrote: >> >> Hi Alex, >> >> sincerely, I don't understand that, and was not what I understand for >> your >> first email. I must say that I completely disagree that we need to do >> 90 or >> 100 commands to get a release that should be around 15. I think we >> have a >> problem with what we really need here. >> >> Maybe the solution is to get some external mediator that have a good >> experience with multiple build systems and releases and could give us >> his >> opinion from outside, so we can have the most logic option. I think if >> we >> can get someone that could give a hand could be very good for us. >> >> Can we ask in Apache for someone that review it? Maybe this will be the >> only solution to avoid going in circles again, since I'm afraid that my >> response will not be what you expect, but sorry, don't see this as you. >> >> I think this shouldn't be a requirement to all RMs, so people that >> wants to >> do all that commands can do it, but I sincerely prefer to stick with >> the >> recipe that is the current standard in all projects (even the ones that >> have more than one build systems). Take for sure that if I see a need >> to do >> all of that I'll be glad to do it. But sincerily is something I don't >> share. >> >> Anyway for people that wants to do that kind of checking I think you >> get a >> good advance from the previous solution. >> >> Thanks >> >> >> >> >> >> >> >> El jue., 2 abr. 2020 a las 21:50, Alex Harui (<[email protected] >>> ) >> escribió: >> >>> Hi Carlos, >>> >>> The command lines in [1] have been added to [2]. [1] is only a >> partial >>> list of the things to do, but includes some helpful setup information >>> Hopefully in June we won't need to release build-tools so you will >> only >>> have about 90 steps to do, but we may have added more commands as we >>> improve our automated test infrastructure. >>> >>> HTH, >>> -Alex >>> >>> [2] >>> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Froyale-asjs%2Fwiki%2FTask-List-For-Royale-Releases&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=V%2BSDjqbnq0yRYrAGvGvVrLRPtdUWy4T0wXidpMx%2FgMU%3D&reserved=0 >>> >>> On 4/2/20, 11:56 AM, "Carlos Rovira" <[email protected]> >> wrote: >>> >>> Hi Alex, >>> >>> to understand your long email. Lets say that when I'll go to >> release in >>> June (0.9.9), I'll use instructions described in [1]. >>> So that will create the sources needed to post. Then, to avouid >> later >>> problems for people verifying I'll verify it with Maven and Ant >> (build >>> with >>> both, and test SDK generated in example apps. Then push to >> dist.a.o, >>> create >>> discuss and vote threads, and start the vote. >>> >>> Is that ok? >>> >>> Thanks >>> >>> [1] >>> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Froyale-asjs%2Fwiki%2FNew-Release-Manager&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=IBIgHvzutG7YtH27kxyD5tyaQyjBe6QetVWWsrSxF9w%3D&reserved=0 >>> >>> El jue., 2 abr. 2020 a las 19:16, Alex Harui >> (<[email protected] >>>> ) >>> escribió: >>> >>>> I'm not sure I understand the distinction. I think we want to >> do >>> both. >>>> The goal of code coverage is to try to exercise paths. We >> want to >>> run "ant >>>> release" because our Ant users might want to do that. And Ant >> does >>> have >>>> assertions AFAICT. It will report errors. Meanwhile, the >> standard >>> for >>>> the .tar.gz package is the one produced by "ant release" >> because >>> that's the >>>> recipe we've been using for years now. The Maven >> distribution's >>> .tar.gz >>>> has been shown to work in most cases, but AFAICT, is nearly as >> well >>> tested >>>> and has not been binary-compared. Ways to compare the two >> .tar.gz >>> files >>>> are needed and welcome. >>>> >>>> More and better tests are welcome. >>>> >>>> Do we have agreement? It sounds like it. So I will now be >> spending >>> my >>>> evenings on the release instead of writing lists of 100 things. >>>> >>>> -Alex >>>> >>>> On 4/2/20, 8:40 AM, "Yishay Weiss" <[email protected]> >> wrote: >>>> >>>> Chris, I think you’re missing Alex’s point. We’re not >> running >>> Ant to >>>> make sure it doesn’t blow up. We’re running it to make sure the >>> resultant >>>> tar.gz/zip files are identical to the ones created by Maven. >> Alex, >>> please >>>> correct me if I’m wrong. >>>> >>>> ________________________________ >>>> From: Christofer Dutz <[email protected]> >>>> Sent: Thursday, April 2, 2020 4:42:00 PM >>>> To: [email protected] <[email protected]> >>>> Subject: Re: Royale Releases >>>> >>>> And to add a little to that [1] >>>> >>>> "In computer science, test coverage is a measure used to >>> describe the >>>> degree to which the source code of a program is executed when a >>> particular >>>> test suite runs." >>>> >>>> So no test, no coverage. Just using something and it's not >>> blowing up >>>> isn't a test for me. It's better than nothing however. >>>> >>>> Chris >>>> >>>> >>>> >>>> [1] >>>> >>> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCode_coverage&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=ohLzW9qtDjCrsy8mHX47Z2qkWR4me5giS%2BSEGSUSWKQ%3D&reserved=0 >>>> >>>> Am 02.04.20, 15:39 schrieb "Christofer Dutz" < >>>> [email protected]>: >>>> >>>> Hi folks, >>>> >>>> I would say we have coverage for both maven ant >> equally as >>> we're >>>> building the same code. >>>> >>>> However we are missing the important assertions. It's >> not >>> that the >>>> Ant build is running some tests Maven isn't. >>>> It's just that the settings for Ant seem to be >> different >>> than for >>>> Maven and the Ant ones happen to work. >>>> >>>> Ideally there would be real tests that test the output >> of >>> both to >>>> see if it works in both cases. >>>> >>>> Chris >>>> >>>> >>>> >>>> Am 02.04.20, 15:15 schrieb "Harbs" < >> [email protected]>: >>>> >>>> Adding more coverage for Maven is good. >>>> >>>> Removing coverage for Ant is not. >>>> >>>> Do you agree? >>>> >>>>> On Apr 2, 2020, at 4:07 PM, Carlos Rovira < >>>> [email protected]> wrote: >>>>> >>>>> Hi Harbs, >>>>> >>>>> I think what we're trying to say is that until >> now we >>>> released with Maven >>>>> and Ant, and that was hiding a flaw in Maven (SVG >>> example). >>>> So that means >>>>> what we were trying to cover was not covered >> clearly, >>> so the >>>> premise is not >>>>> right. >>>>> >>>>> >>>>> >>>>> El jue., 2 abr. 2020 a las 14:56, Harbs (< >>>> [email protected]>) escribió: >>>>> >>>>>> No one is arguing that we shouldn’t add more >> tests. >>>>>> >>>>>> Please let’s not make it seem like there’s a >>> disagreement >>>> about that. >>>>>> >>>>>>> On Apr 2, 2020, at 10:46 AM, Carlos Rovira < >>>> [email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi Alex, >>>>>>> >>>>>>> first, many thanks for the detailed email. I'll >>> comment on >>>> this later as >>>>>> I >>>>>>> have more time. >>>>>>> >>>>>>> For now, to add up a recent example on what >> Chris >>>> commented: If you all >>>>>>> remember a week ago I was trying to use SVG >> Images >>> in a >>>> blog example that >>>>>>> was published 2 days ago. Nobody tried SVG >> Images >>> before >>>> building with >>>>>>> maven, I know that since maven was not properly >>> configured >>>> and using that >>>>>>> component from Maven was failing with an RTE. >>> Probably we >>>> have more >>>>>> things >>>>>>> not working the same way when build from Maven >> and >>> Ant, >>>> and that's >>>>>>> something that will need people using that code >>> paths in >>>> test >>>>>> applications >>>>>>> (or in their own apps) to see if things works >>> properly. >>>>>>> >>>>>>> I was recently introduced to >>> "examples-integrationtest" by >>>> Chris, that I >>>>>>> plan to use soon as I can. I think is a great >> idea, >>> since >>>> you get a >>>>>> Firefox >>>>>>> running test interface of the real use of some >>> concrete >>>> royale code. I >>>>>>> think passed until now unnoticed by all of us, >> and >>> seems a >>>> powerful tool. >>>>>>> There's already an example about FlexStore >> with some >>> basic >>>> assertions. >>>>>>> >>>>>>> Again, thanks, and will comment on the rest >> later >>>>>>> >>>>>>> Carlos >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> El jue., 2 abr. 2020 a las 9:20, Christofer >> Dutz (< >>>>>> [email protected]>) >>>>>>> escribió: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> just a point you are bringing up: "Code >> coverage". >>>>>>>> >>>>>>>> I strongly dislike the idea of "asjs" >> effectively >>> being >>>> the test for the >>>>>>>> compiler. The reasoning behind this is: yes >> you do >>> get >>>> more code >>>>>> covered, >>>>>>>> but only the happy-path (ideally) and even if >>> things go >>>> wrong, the end >>>>>>>> results aren't tested. Did add a module to >> asjs >>> years ago >>>>>>>> ("examples-integrationtest") that deployed the >>> examples >>>> in a tomcat >>>>>> server >>>>>>>> then opens a Firefox browser and clicks >> through 2 >>> of the >>>> examples (I >>>>>> added >>>>>>>> two dummy tests as an example, but seems no >> one >>> touched >>>> this after me). >>>>>> I >>>>>>>> did this because I remember us working on >> asjs for >>> weeks >>>> without anyone >>>>>>>> noticing the compiler wasn't producing >> runnable >>> code ... >>>> same with the >>>>>>>> little unit-tests that are still run for every >>> example, >>>> that simply >>>>>> check >>>>>>>> if an output is generated, because we had a >>> prolonged >>>> period of time >>>>>> where >>>>>>>> we were all working on different parts, but >> for >>> quite >>>> some time the >>>>>>>> application compilation just didn't output >> anything >>> and >>>> no one noticed. >>>>>>>> >>>>>>>> So coverage is nothing without assertions (my >>> opinion) >>>> ... ok ... it's >>>>>>>> slightly better than no coverage, but not >> much, in >>> my >>>> opinion. >>>>>>>> >>>>>>>> I think in parallel to this release >> discussions I >>> have >>>> seen numerous >>>>>>>> threads about someone doing something that >> broke >>>> something for someone >>>>>>>> else. This could be addressed by increasing >>> coverage by >>>> providing >>>>>> explicit >>>>>>>> tests. >>>>>>>> >>>>>>>> Coming back to the releases: >>>>>>>> I have no objections, if you do a "release" >> locally >>> and >>>> automate the >>>>>>>> validation on the CI server (Which effectively >>> would be >>>> your proposal >>>>>> to do >>>>>>>> the first 12 steps on local hardware and the >> 13th >>> on the >>>> CI server). I >>>>>> even >>>>>>>> think that's a good idea ... There could be >> one >>> step for >>>> building a >>>>>> release >>>>>>>> from a given "git tag" for every build system >> and >>> generic >>>> means to >>>>>> compare >>>>>>>> tar.gz and zips produced by any build system >> with >>> that of >>>> another >>>>>> (ideally >>>>>>>> with better output than just a plain >> "true/false"). >>> This >>>> would even >>>>>> help to >>>>>>>> iron out the last potentially existing bumps >> out of >>> the >>>> Maven >>>>>> distribution. >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Am 02.04.20, 07:59 schrieb "Alex Harui" >>>> <[email protected]>: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> This is my attempt to explain what goes >> into a >>> release >>>> in hopes that >>>>>>>> we can understand and agree on what our >> release >>> process >>>> is. It became >>>>>>>> apparent in my reading of the wiki page with >> the new >>>> Maven steps and in >>>>>>>> talking with Harbs today that there are still >> many >>>> misunderstandings >>>>>> about >>>>>>>> what we do to create a release. I don't >> generally >>> like >>>> writing >>>>>>>> instructions in English because it is easy to >> be >>>> ambiguous. All of the >>>>>>>> steps that we use to create releases had been >>> captured in >>>> Ant scripts >>>>>> in a >>>>>>>> much more explicit way, IMO, but I took the >> time to >>> write >>>> them down in >>>>>>>> English here: >>>>>>>> >>>>>> >>>> >>> >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Froyale-asjs%2Fwiki%2FTask-List-For-Royale-Releases&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=V%2BSDjqbnq0yRYrAGvGvVrLRPtdUWy4T0wXidpMx%2FgMU%3D&reserved=0 >>>>>>>> >>>>>>>> I did this quickly by scanning the CI >> steps, the >>> new >>>> Maven steps and >>>>>>>> the Ant scripts used in prior releases so >> there >>> could >>>> certainly be >>>>>> mistakes >>>>>>>> and missed steps. If I did my math right, >> the RM >>> for >>>> 0.9.7 will have to >>>>>>>> complete over 100 tasks (essentially, typing a >>>> command-line 100 times). >>>>>>>> Future RMs, when we don't have to release >>> build-tools, >>>> will have about >>>>>> 92 >>>>>>>> steps. And I did not include voter >> verification >>> checks >>>> the RM should >>>>>> run >>>>>>>> before opening a vote (verifying that the >> artifacts >>>> download and match >>>>>>>> their checksums, etc). As an RM, I run a >> bunch of >>> tests >>>> on the RC >>>>>> before >>>>>>>> sending out the vote. Maybe we should add >> those to >>> the >>>> task list. >>>>>>>> >>>>>>>> I think there has been confusion about the >> use of >>> Ant >>>> in the release >>>>>>>> process. Because I was the RM for the first >> set of >>>> FlexJS/Royale >>>>>> releases, >>>>>>>> and I'm a lazy person who hates typing at the >>> command >>>> line, I created >>>>>> Ant >>>>>>>> scripts to execute these 100 steps. But I >> agree >>> that it >>>> is not a >>>>>>>> requirement that other RMs must use the Ant >> scripts >>> for >>>> these >>>>>> commands. If >>>>>>>> you are the RM and like typing, go ahead. >>>>>>>> >>>>>>>> Then we found out that other people >> couldn't get >>>> through this task >>>>>>>> list. I think the 3 people who tried were >> having >>> trouble >>>> with Maven >>>>>>>> uploads and downloads. So what I did was put >> the >>> first >>>> 40 steps or so >>>>>> into >>>>>>>> Jenkins jobs. And by doing that, Piotr was >> able to >>>> produce our last >>>>>>>> release. And that also saves on manually >> typing >>>> commands. But again, >>>>>>>> going forward, the RM gets to choose how they >> want >>> to >>>> execute these >>>>>> steps. >>>>>>>> >>>>>>>> If you scan the set of steps, you'll see >> that >>> "ant" is >>>> only in there >>>>>>>> once. I believe the recent threads have been >> about >>> this >>>> single command >>>>>> out >>>>>>>> of the 100+ commands. This is why this has >> been so >>>> frustrating to me. >>>>>> I >>>>>>>> believe there is a solid technical reason for >> that >>> one >>>> command: it >>>>>> proves >>>>>>>> that the build.xml files in the source >> packages can >>> build >>>> the .tar.gz >>>>>> that >>>>>>>> are useful to NPM and IDE users who use Ant >> and >>> want to >>>> test a change. >>>>>>>> >>>>>>>> I think of it as code-coverage. If we had >>>> code-coverage tools, we >>>>>>>> would ask that the RM complete as much of the >>> automated >>>> code-coverage >>>>>>>> testing as possible before posting the >> release for a >>>> vote. That one >>>>>>>> command increases our code coverage by >> running the >>>> build.xml files. We >>>>>>>> should be always working to increase >> automated code >>>> coverage in the RC. >>>>>>>> Certainly for me as RM, I will gladly watch >> TV as >>> the >>>> automated tests >>>>>> run >>>>>>>> because a failed RC means going back through >> many >>> of the >>>> first 25 >>>>>> commands >>>>>>>> again and wastes other people's time. Each >> RC is >>> more >>>> emails to read >>>>>> and >>>>>>>> more time from the voters and testers. >>>>>>>> >>>>>>>> If there are other ways for the RM to get >> the >>> same or >>>> better code >>>>>>>> coverage on the build.xml files before >> posting the >>> RC, we >>>> can discuss >>>>>> those >>>>>>>> options. >>>>>>>> >>>>>>>> I am hopeful we can all agree on these >> simple >>>> principles: Strive for >>>>>>>> better code coverage and fewer failed RCs. >>> Royale's main >>>> purpose is to >>>>>>>> save other people time. Let's do that in >> creating >>>> releases too. >>>>>>>> >>>>>>>> One issue that was brought up recently was >>> whether it >>>> is a good >>>>>>>> decision to have the RM test all of the build >>> platforms >>>> we support. >>>>>>>> Suppose we add some other build system >> support or >>> more in >>>> the future? >>>>>>>> Again, the code coverage principle applies >> here, but >>>> also, I would like >>>>>> us >>>>>>>> to retain feature parity, and I also hope for >> as >>> few RCs >>>> and votes as >>>>>>>> possible. So instead of having separate >>>> votes/features/release-dates >>>>>> for >>>>>>>> the Maven artifacts vs the Ant artifacts vs >> the >>>> SomeFutureBuildTech >>>>>>>> artifacts, I think we should have one vote >> and keep >>> them >>>> all in sync. >>>>>> If >>>>>>>> we do ever get around to monthly or bi-monthly >>> releases, >>>> I think >>>>>> separate >>>>>>>> build platform releases would be too much >> work. >>>>>>>> >>>>>>>> But consider this thought I just had >> today: the >>> RM >>>> doesn't really >>>>>>>> have to choose to do all 100 commands on a >> local >>> machine >>>> or with Ant >>>>>>>> scripts or do the first 40 via CI. The RM can >>> actually >>>> pick and choose >>>>>>>> commands to run on the CI server. The CI >> Jenkins >>> jobs >>>> are not a >>>>>>>> separate/alternative release process, they >> are just >>>> another way of >>>>>>>> executing the first 40 steps. Using CI jobs >>> actually >>>> requires >>>>>> additional >>>>>>>> command-line cut-and-paste to push commits on >> the CI >>>> server and to sign >>>>>> and >>>>>>>> validate binaries locally, but that's the >> trade-off >>> of >>>> not having to >>>>>>>> configure your machine to successfully run >> all of >>> the >>>> automated tests >>>>>> and >>>>>>>> build systems, and being able to run a >> command by >>> filling >>>> in the version >>>>>>>> number and rc number and hitting the "ok" >> button >>> instead >>>> of making sure >>>>>> you >>>>>>>> got the whole command typed in correctly. >>>>>>>> >>>>>>>> So, an RM can run the first 25 steps >> locally, >>> then go >>>> the CI server >>>>>>>> and run what is now Jenkins Job >>> "Royale_Release_Step_013" >>>> (no need to >>>>>> run >>>>>>>> the first 12) and it will run tasks 26 >> through 32, >>> and if >>>> it is >>>>>> successful, >>>>>>>> then the RM has proven code coverage of the >>> build.xml >>>> files. (If the >>>>>>>> resulting tar.gz and zips are not posted, >> then the >>> RM >>>> should verify that >>>>>>>> they match the ones from Maven >> distribution). I >>> would >>>> encourage RMs to >>>>>>>> also use the CI jobs that generate the emails >> to >>> make >>>> sure the subject >>>>>> and >>>>>>>> content is correct and contains the usual >>> instructions so >>>> we have >>>>>>>> consistency. Maybe someday there will be CI >> jobs >>> to do >>>> the last 60+ >>>>>> steps >>>>>>>> if that helps. We could add a Jenkins job >> that >>> runs an >>>> Ant build on RC >>>>>>>> artifacts on dist.a.o as well. >>>>>>>> >>>>>>>> I would like you all to help maintain the >> list of >>> 100 >>>> steps and other >>>>>>>> documents related to the release process, and >>> improve the >>>> CI jobs and >>>>>> Ant >>>>>>>> steps if it helps you be a more efficient >> RM. I am >>>> hopeful that now >>>>>> that I >>>>>>>> have hopefully explained our release process >>> better, that >>>> we can see >>>>>> that >>>>>>>> these 100+ steps just have to be done in some >> way. >>> The >>>> RM can figure >>>>>> out >>>>>>>> what way works best for them, but they must >> get >>> through >>>> all of them. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> -Alex >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Carlos Rovira >>>>>>> >>>> >>> >> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=TH74uNC5GsYB4%2BS3d2SCU4Ckwi8WbszDDTVXurLfAqc%3D&reserved=0 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Carlos Rovira >>>>> >>>> >>> >> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=TH74uNC5GsYB4%2BS3d2SCU4Ckwi8WbszDDTVXurLfAqc%3D&reserved=0 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> Carlos Rovira >>> >>> >> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=TH74uNC5GsYB4%2BS3d2SCU4Ckwi8WbszDDTVXurLfAqc%3D&reserved=0 >>> >>> >>> >> >> -- >> Carlos Rovira >> >> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Ccdfb6a3f66b34452025f08d7d75cc39d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637214665746612222&sdata=TH74uNC5GsYB4%2BS3d2SCU4Ckwi8WbszDDTVXurLfAqc%3D&reserved=0 >> >> >> > > -- > Carlos Rovira > http://about.me/carlosrovira
