If I recall correctly, one the goals of using the CI was to enable incremental release so more than one contributor could work on a single release. I really hope that can be achieved whatever path is chosen.
From: Carlos Rovira<mailto:[email protected]> Sent: Wednesday, March 18, 2020 2:09 AM To: Apache Royale Development<mailto:[email protected]> Subject: Re: About release 0.9.7 and how to fix and streamline the process Hi Alex, as I see it we have mostly two ways to choose: 1) Left Chris and I try to fix Maven distribution SDK (compilation with IDE is the only thing broken). And try what we defend. Maybe we can be wrong. I don't mind to recognize it if we reach a wrong path (Maybe in this process we find some low level issues that we didn't know and fail in the attempt. It's clear that we don't know as you all the involved things in the process, and if that happens, will be easy to say, "Hey! the proposal was not ok since we found X and Y, so that can't be done"). In the other hand, if we succed...We (all) can get from this effort a more easy way to release. We know we need to ensure Maven and Ant continue to work as a build and can be reproducible. I think that must be the main point for all of us and not any other thing. Just that. If we get success along the way we get some improvements and streamline the process, I think we all will benefit from that and Apache Royale as a project will benefit of all the improvements done with that effort. All of this of course taking into account that I can get Chris help to do this. Note for Harbs: If we can ensure ANT build is working right (I assume that is what you should care since is the build system you use), where is the point on imposing us the need to complicate a release with doing such intricate process? 2) We continue with the CI steps. But for this I'll want to continue improving the Maven build process and that should not be a problem for CI steps. For this I can try to help on fixing CI steps. If you remember I was asking here questions, and as well some messages from the CI was sent for steps 1,2 and 3 during some days when I tried to go over it, but with no success at all. I even create duplicates of CI steps on the CI to ensure I work over copies to avoid change things over main ones and lost old ones. Maybe Harbs, Yishay or others can try to fix it and see if they find it easy and reliable for the future. Their experience will be very valuable. My experience was not good, but maybe I'm wrong and other's experience is better. I think Piotr's was hard, since he spend almost 2 weeks in this try, but I think he finally was ok. Alex to respond to some other points in you email and conclude: * for "industrial" I mean a proven way that other software projects use to release. Almost all projects here at Apache seems to release in a standardized way and all uses Maven, so to me that's what I call an "industrialized build process" that ensures a results, stability and reliability as well as easy to update as changes in the project evolve over time * My perception of the problems other RMs had over time with building in its own machines, or upload artifacts to a.o and other issues was caused for our current not standardized (not industrial) build process that mix Maven and Ant. So by choosing 1) I expect those problems to be removed and release without effort from a normal machine. I'm as well only just "one vote" as you. We saw already what Harbs thinks. Maybe others should express as well, even if they are not PMC since I think this is important. As I said this is for me maybe the most important problem we have right now, and is clear its a problem since we didn't release since October, 2019. And right now, we can't because the CI steps are broken and maybe just you know how to fix it, and you already stated several times that you don't plan to do it. Chris and I can try, but in other different way that we think is more "standard" or "industrial". So others must pronounce to see as a community what we should do, taking into account that we need something that needs to be reliable over time and not break with each improvements we want to do to build scripts (Maven or Ant), and make easy for a RM to do the task without spending a several days on it. to conclude, if people finally want to go with CI process (2), I think I'll try to ask Chris for doing the maven improvements, we wanted as soon as we can, so then others can try to go over CI steps and see if they are more lucky than me on trying it, just to avoid fix, change maven, and need again to fix. Thanks El mar., 17 mar. 2020 a las 22:26, Alex Harui (<[email protected]>) escribió: > HI Carlos, > > I only have one vote. But we've spent a lot of time on this issue already > and the answer has always been the same: use Maven for Maven artifacts, use > Ant for Ant artifacts. I would much rather you spend your energy on fixing > what was a working release process. Of course it didn't work perfectly for > Piotr at first because it is a complex process, but eventually it did > work. If we did change to some new process that you want should we have > the same strict criteria that if it doesn't work the first time it must > therefore be no good? I'm sure you would not agree to that. > > I don't know what you mean by "industrial", but I'm quite certain these > days that we can't use the "expected" solution. This is a key point. You > and Chris have not been through the release process so you don't know the > kinds of issues that have to be resolved. I explained these issues in my > earlier response. I used a much more "expected" solution for the first > Royale releases and we found out that nobody else except me could use > them. So we were forced to try this CI multi-step process and finally, we > had someone be a successful RM other than me. So, any solution that might > work for you and Chris that uses Maven in the "expected" way on your > computer is likely to fail to work for other people. That's just our > reality. > > I do not recall seeing any questions about the CI steps since the new > Maven build seemed to stabilize, so I don't understand why you chose to > spend so much time on fixing them without asking a question on our lists. > I will try to answer questions if you have them. The steps are pretty > simple: Whatever steps you would normally run on the command-line to > create a release are captured in these steps. At some point, the steps > generate something that needs to be committed (like a branch, or updated > poms) or signed, and so that Jenkins job ends so the RM can log into the CI > server and commit the changes, then another step continues on until the > next commit or signing is needed. > > I will try to find time to understand how the Maven build now works, but I > am going to try really hard not to be the person who goes and fixes these > release steps. Not only did I not break them, but it would be better for > others to understand more about how they work. There is certainly room for > improvement in the steps, but fundamentally, we have to have a process > where some shared computer with a known good configuration does some work, > the RM then does a commit or sign, and so on. > > There are no restrictions I can think of on what Maven can be tasked to do > within a step, the only restriction occurs when something needs to be > committed or signed. And we need reproducible binaries in order to > validate the artifacts. Once you have local computer validation it makes > sense to me to use Maven's Wagon to upload to repository.a.o instead of the > Maven release plugin, but there might be a better way. > > In the end, though, Maven will build the Maven artifacts and Ant will > build the Ant artifacts. That way we validate as we build the artifacts. > > My 2 cents, > -Alex > > > > > > On 3/17/20, 12:50 PM, "Carlos Rovira" <[email protected]> wrote: > > Hi Alex, > > I don't think you should be upset, don't think that to be a positive > way to > discuss this task, since in that way you force a solution and maybe > not all > of us think the same way as you. So although it could cause some > tiring to > all of us (me included) I think we need to put all our effort to get to > some point where we all be satisfied. > > This is probably the most problematic task we have in Royale, and > since is > difficult to solve and seems there are at least two groups of people > thinking in different ways, we need to get what's the better way to > solve > it. No one way of thinking is better that other you have valid points > (but > in the same way I can find arguments against them), but don't think > that is > a productive route. I want to spend our times in get a real solution. > > The CI steps you did was a great effort and very appreciated. I was > very > happy when you made it, since from outside it all sound very promising. > Then IIRC Piotr's release doesn't was as straight as expected. So I > think > that was a point we should all take into account. Then my experience > updating maven and broken some steps was as well something from what we > should learn and think about it. I tried to fix them without exit. I > still > want to improve Maven even more in the future. One of the problem I > see is: > If we fix CI steps, will broke again when new changes in maven will be > done? probably...and will the fix as difficult as now? In my > experience, > probably too. > > Don't think as well that is good for this discussion impose a way to > act. > I'll be fixing CI steps if I could, but I repeat that I tried during > some > days without exit. Spend many hours over it. So is not something I'd > not > do, is just that the process is very fragile and is not what I expect > it > will bring to us. Each thing we change can break it and that's a huge > problem ahead. Or is just me the only one that see a huge problem on > this?. > > Instead, I can offer my work in fixing in other way that I think is > more > "industrial" or proven, but you don't like it, due to the points you > express (that I repeat can be valid, but I see that all are totally > surmountable If we work in a more industrial way to solve it. Don't > think > we are more special that other hundreds of projects out there, and all > of > that projects just use maven for release. > > Maven SDK could still not be 100%, that's right, but is one of the > things > we need to fix in the actual process I offer. For sure. And releasing > that > way, you still can build with ANT and check it was all right. We don't > need > to do at the same time that releasing. I think that's a valid point. > > I think that if you defend the CI steps in such fixed way, the rest of > us > thinking that's not the way to go can just not oppose, but you can't > ask us > to be responsible of trying to solve something that is very (very) > difficult to solve, since maybe we just can't do it. You must take into > account that new changes could come (to ant, to maven), and working > that > way should be your responsibility or the any others here that think in > that > way. I can help or support in the way I can, but maybe limited in some > way. > And of course, we should all know that new optimizations to build > systems > can't be stopped to defend a concrete way to release. IOW, release is a > process to "publish for other", and we can't block progress in the > project > or in the way we build to support a concrete way of release. > > just my 2 > > Carlos > > > > > > El mar., 17 mar. 2020 a las 19:50, Harbs (<[email protected]>) > escribió: > > > +1 > > > > Harbs > > > > > On Mar 17, 2020, at 8:10 PM, Alex Harui <[email protected]> > > wrote: > > > > > > I’m sorry, but these repeated attempts to try to use Maven too > much just > > make me very upset. This was a concern of mine when two people who > have > > never produced a release went off to refactor the process. There is > theory > > and reality. The reality is that Maven cannot and should not be > used to > > streamline our release process because: > > > > > > - We've learned over the years that RMs have trouble getting set up > > correctly. > > > - We've learned over the years that RMs have trouble uploading > Maven > > artifacts to repository.a.o. > > > - We should know that the only way to validate the Ant Task in the > > release artifacts is to use Ant to verify it. > > > > > > Those are the three main reasons there are multiple release steps > on the > > CI server. I am hopeful that the new Maven builds can cut down on > one or > > two steps, or maybe just make those steps more straightforward, but > the > > process simply cannot just run Maven's release steps on the RM's > local > > computer. It must run on the CI server, and that requires stopping > from > > time to time to verify and sign stuff. > > > > > > A fourth reason was that the Ant script had proven to be able to > build > > an IDE-compatible artifact. I am not 100% convinced that the Maven > > distribution build can exactly replicate that artifact. But again, > if an > > Ant user wants to use Ant to create artifacts for some IDE to test > local > > changes, then we need to prove in each release that Ant can produce > that > > set of artifacts for the IDE. So now that Maven can supposedly do > the > > same, it actually adds more work to the verification process. The > Maven > > artifacts for an IDE will need to be compared against the > Ant-produced > > artifacts for an IDE. > > > > > > In short, we should use Maven to produce Maven artifacts, and Ant > to > > produce Ant artifacts so that the RM is verifying the artifacts long > before > > the voters have to. And make it all work on a CI server so that > RM's don't > > have to deal with system setup, and not have upload issues blow up > the > > process. > > > > > > I also am not a fan of using Maven to do scripting, especially if > it > > involves custom mojos/plugins. That's what scripting tools like Ant > are > > for. The goal should be to make the release system maintainable by > > everyone, not just by experts in Maven. Similarly, the Ant steps do > not > > use custom plugins/tasks for the same reason. > > > > > > Use Maven to build the Maven artifacts. Use Ant to build the Ant > > artifacts and sequence the steps. Use the CI server. Verify and > sign > > locally. These are the requirements. It was all working and we > could have > > released every month. I think it is your responsibility to make it > work > > again within those constraints. > > > > > > My 2 cents, > > > -Alex > > > > > > On 3/17/20, 8:48 AM, "Carlos Rovira" <[email protected] > <mailto: > > [email protected]>> wrote: > > > > > > Hi > > > > > > this morning in the slack Apache Royale official channel Chris > Dutz > > asked > > > about plans for release 0.9.7. If you see Greg and I help him to > > create an > > > Apache Royale interface for pulling data from a Consumer in an > IoT > > Apache > > > PLC4X app. Something very cool :). He published a tweet with a > video > > > showing it. > > > > > > So regarding the release, I explain that I spent many days some > > months ago > > > trying to fix the CI release steps to release, but I was unable > to do > > it. I > > > think making changes to the Maven o Ant make the 13 steps very > > fragile and > > > difficult to fix. And we need to evolve continually Maven and > Ant, so > > that > > > means a huge task that will require an huge amount of time that > almost > > > nobody here has. So I must say that although the work done on > that > > front > > > was amazing and huge maybe we need to streamline it to an > standard. I > > know > > > is difficult, but although we had many expectations put on the > 13 step > > > release process, after try it seems that we finally are not > getting > > what we > > > wanted, that is streamline the release process. > > > > > > This is one of the critical points that hold us to release 1.0, > since > > not > > > having an easy way to release means as we push 1.0 and people > tries > > us, > > > we'll need to be faster on releasing, and we can't wait 6 month > or a > > year, > > > since that we'll mean user will not want that kind of release > cycle. > > > > > > So here is my proposal to work on a release process that allow > us to > > > release more often and more easily. Hope you all can consider > it: > > > > > > First: We all know we want build with ANT and MAVEN. And when I > refer > > to > > > "all" I really mean "all". Although I'm a Maven advocate, I > really > > > appreciate the point to have ANT on place so having 2 build > systems > > that > > > completely works mean advantage. In fact although I use maven I > build > > with > > > both system daily. > > > > > > Second: Build does not mean Release. I think other times we > discussed > > > Release as a way to impose Maven over Ant. Or maybe that was the > > perception > > > for others, but not my intention (and maybe not the intention of > > others). > > > But that's not the objective. Release [1] means to us just > > "publication > > > outside the development community, defined as individuals > actively > > > participating in development or following the dev list." > > > > > > That means that we don't need to release using Maven and Ant at > the > > same > > > time, we can streamline greatly the process to just releasing > with > > Maven so > > > we can ensure a simple process the same as any other Apache > project. > > > > > > It means that people trying the release need to check it with > Maven, > > but > > > community users can continue using ANT or Maven to build or > even just > > > download the SDK and use it on his IDE of choice and don't use > ANT or > > Maven > > > at all. > > > > > > Third: The fact is the rest of projects I know all use releases > and CI > > > based on Maven, and that should point us that maybe we're > trying to > > > force something that is not standard in our industry, so if we > want > > to make > > > it more reliable and easy process, is normal to embrace how > others are > > > doing, since that does not mean we're throwing away a build > system, > > since > > > per previous point we don't want to remove ANT. > > > > > > Finally, Chris and I offer our time to make the easiest release > > process we can > > > that let all of us release easily and often (maybe each 1-2 > month?) > > > > > > Thanks > > > > > > > > > [1] > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23release-definition&data=02%7C01%7Caharui%40adobe.com%7C525c830261c347c9dc8d08d7caac68a3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637200714153362334&sdata=49KtjLtJDXsCKMMrIEr2W7zEt5L%2FTNk6Rtvl9pgpBBo%3D&reserved=0 > > < > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23release-definition&data=02%7C01%7Caharui%40adobe.com%7C525c830261c347c9dc8d08d7caac68a3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637200714153362334&sdata=49KtjLtJDXsCKMMrIEr2W7zEt5L%2FTNk6Rtvl9pgpBBo%3D&reserved=0 > > > > > > -- > > > Carlos Rovira > > > > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C525c830261c347c9dc8d08d7caac68a3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637200714153362334&sdata=PP%2Bq0O86F%2FguQtSIDzpGOa3CIM4qTiKmf%2FlmUVS7%2B%2Bo%3D&reserved=0 > > < > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C525c830261c347c9dc8d08d7caac68a3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637200714153372330&sdata=t0BWYYpiCDSPpjPtrRG7mfM%2F%2Bodl0zf4a3j%2B591gFJA%3D&reserved=0 > > > > > > > > -- > Carlos Rovira > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C525c830261c347c9dc8d08d7caac68a3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637200714153372330&sdata=t0BWYYpiCDSPpjPtrRG7mfM%2F%2Bodl0zf4a3j%2B591gFJA%3D&reserved=0 > > > -- Carlos Rovira http://about.me/carlosrovira
