FWIW, the code attributed to me in SWCWriter seems to also include a change made two days ago that was not made by me (specifically, the call to setTimeZone). So the original attempt by me did not set a timezone at all. The TODO comment was also not written by me.
After doing more digging, I came across a couple of hopefully relevant web pages: https://stackoverflow.com/questions/27675454/zipentry-gettime-unpredictable-results https://www.baeldung.com/java-daylight-savings If I understand the first web page correctly, the Zip file does not contain some standard like "milliseconds since the epoch", but rather, a unique encoding into a long based on just the Year, Month, Day, Hour, Minute, Second. No milliseconds or timezone information. If I understand the second web page correctly, because we are using Java 8, when the Zip library converts the "milliseconds since the epoch" we pass in into this unique encoding, it might produce the wrong results. I haven't taken the time to research the actual format in the Zip file, but if the encoding converts the "milliseconds since the epoch" back into a Date or String and then does something like: Long timestamp = date.year << 28 + date.month << 21 + date.day << 16 + date.hour << 12 + date.minutes << 6 + date.seconds; Then whenever the Java compiler version on the computer creating the SWCs has out-of-date information about the whether "daylight savings time (DST)" a.k.a "summer time" should be to used to convert the "milliseconds since the epoch" back to a Date or String then I think you'll get incorrect results. There may be a way to tell Java 8 to use different DST change dates. But it may be that the original code I put in is as good as we can do in our code. Of course, I could be wrong... -Alex On 3/25/20, 1:39 PM, "Alex Harui" <aha...@adobe.com.INVALID> wrote: Reproducible builds for Royale is certainly new. There could be bugs in North vs South hemispheres. Don't know. Volunteers are welcome to improve this portion of our code. I believe one (maybe the only one) of the challenges is to figure out how to choose a timestamp that can be used in the SWCs for at least those needing to reproduce the SWCs. The ZipEntry used in the SWC does not contain a timezone so I think there are some challenges to that. You can try building the SWCs using the timestamps used in the daily builds and see if you also see time skews and propose solutions. -Alex On 3/25/20, 1:28 PM, "Greg Dove" <greg.d...@gmail.com> wrote: 'It could be an issue when the US and Europe have different daylight savings time settings..... When we saw it work, we were not at one of those transition points.' Southern hemisphere and northern hemisphere differences pivot by 2 hours through these transition gaps, which are typically 2 weeks or 1 week when the application or non-application of DST is aligned between the two countries being compared. For the majority of the year DST is not aligned. The end of each transition results in things being 2 hours different from whatever was 'normal' before. If the nature of this problem is as-described, which seems to require DST alignment, does that mean the timestamp issues might be present most of the time for someone south of the equator and perhaps only work in the 'transitions' when DST 'settings' are briefly aligned? So far, I know this hasn't been a problem, with Royale having had no RM from southern climes, iirc. So I'm just asking mainly to clarify my understanding of the problem itself. If the answer is "I'm not sure, try it and see" I am not at that point yet :) If, however, there is something specific that I can investigate about timestamp issues, I am happy to try. On Thu, Mar 26, 2020 at 7:23 AM Alex Harui <aha...@adobe.com.invalid> wrote: > That has nothing to do with the Ant build of release artifacts. That is > just using Ant to run command line commands to download and verify Maven > release artifacts. I'm pretty sure you can just look at the steps and type > them in manually at the command line and you'll get the same results. > > AFAICT, the problem is that the reproducible binaries generated by Maven > aren't reproducible. It could be an issue when the US and Europe have > different daylight savings time settings. When we saw it work, we were not > at one of those transition points. So maybe it is best to just wait until > Sunday. But it did work for me in the US and Piotr in EU. > > Proposals for better ways to have an RM verify artifacts built on a remote > machine are welcome. Testing reproducible binaries was the only way I > could think of and implement. > > -Alex > > On 3/25/20, 11:13 AM, "Carlos Rovira" <carlosrov...@apache.org> wrote: > > Hi Alex, > > after run CI Server 007 we get this email: > > From the royale-typedefs repo: > 1. Run ant -f releasesteps.xml Release_Step_007 -Drelease.version=0.9.7 > -DskipTests=true > This will download the artifacts then unzip and compile the source > artifact. > 2. Validate that the compiled artifacts match the downloaded artifacts. > 3. If they do, then run ant -f releasesteps.xml Release_Step_007_Sign > -Drelease.version=0.9.7 > This will PGP sign the source ZIP and compiled JARs > 4. Then run ant -f releasesteps.xml Release_Step_007_Upload > -Drelease.version=0.9.7 > This will upload the signed artifacts to Maven Release Staging. Do not > "Close" the staging repository until the other repos have been added. > > This is what is failing from sub step 007-1, so as part of the step, > I'm > downloading with ant and building, signing and then uploading (Or steps > should allow me to do that, but is failing) > > A part from that I don't think we really want a solution where of > breaking > steps event more. What we're requesting is remove steps. > > > > > El mié., 25 mar. 2020 a las 18:54, Alex Harui > (<aha...@adobe.com.invalid>) > escribió: > > > I'm sorry, I must have missed something. Where was Ant involved in > any of > > the problems reported so far? AFAICT, it was all Maven plugins > driving our > > compiler and we couldn't get reproducible binaries. > > > > I suggested in one reply that Chris fills a staging repo using his > local > > machine just to make sure we know what Maven commands need to be run > to > > generate all of the Maven artifacts. And that would validate the > claim > > that it is really doable in two or three steps. A partial > verification of > > validity is to check that the same set of files was produced as in > 0.9.6 > > (there might be some new SWCs though). > > > > Then someone else (maybe Carlos) would download all of the > artifacts, run > > the same steps, download the new artifacts and verify that the binary > > match. Then we can work on splitting those steps into chunks for > the CI > > server. > > > > -Alex > > > > > > On 3/25/20, 10:13 AM, "Carlos Rovira" <carlosrov...@apache.org> > wrote: > > > > We imposed ourselves a need to release building with two build > systems. > > That's the biggest issue and that's a requirement of this PMC. > > That's what makes the need of the actual CI steps and the > complex and > > intricate process. > > > > For the rest of Apache a release is: > > > > "For an *Apache* project, that means any publication outside the > > development community, defined as individuals actively > participating in > > development or following the dev list. More narrowly, an official > > *Apache > > release* is one which has been endorsed as an "act of the > Foundation" > > by a > > PMC." > > > > That means that we can release with ANT, with Maven, or any > other build > > systems as part of the process, and even try what we're doing of > > trying to > > build with n build systems and try to check all is ok with the > > artifacts > > generated by all that build systems and check all of them...but > if > > that's > > what we want, I personally will left this wagon, and wish you > the best > > of > > lucks in the process. > > > > Chris and I proposed to just use Maven *for release* (so please, > I > > expect > > no body starts to say we are throwing away ANT from the project > for > > building, please....we are just discussing release here). > > > > What do you think about to try to release just with Maven? If > can do > > in the > > next day or two and get the release published...will that work > for > > you? I > > think we don't need anything more, and I'm sure our users will be > > happy too. > > > > And yes, people wanting then to build with Ant or with Maven > will be > > able > > to do so... :-) > > > > Thanks > > > > Carlos > > > > > > [1] > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Fdev%2Frelease-publishing.html&data=02%7C01%7Caharui%40adobe.com%7Cf9c8edc66d194754b80008d7d0fc8f9d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207655475703139&sdata=MDkT1GPQhM5BEqXcqjIHdMt%2FXqQOWDLUnbYFirNHC6c%3D&reserved=0 > > > > > > > > El mié., 25 mar. 2020 a las 17:46, Christofer Dutz (< > > christofer.d...@c-ware.de>) escribió: > > > > > Hi Yiashay, > > > > > > I agree ... but I didn't create Maven ... haven't even > provided a > > single > > > PR to it ;-) > > > I think my only noticeable contribution was that they changed > their > > > website to forbid the usage of "maven-{something}-plugin" to > Apache > > Maven > > > Core modules (It initially said Apache Maven Plugins which > would have > > > applied for maven-flex-plugin and maven-plc4x-plugin) > > > > > > I think you should focus not on the process, but the bases a > process > > > should cover. > > > > > > Focus on what you want to be provided or ensured by any > process, no > > matter > > > which tool it is based on. > > > > > > That's what I created the google doc for. > > > > > > If you decide to loosen the requirement on sticking 100% to the > > existing > > > process I am more than willing to help. > > > However if you stick to the current process, it doesn't make > any > > sense for > > > me to do so. > > > > > > Looking forward to a productive discussion, > > > > > > Chris > > > > > > > > > Am 25.03.20, 17:36 schrieb "Yishay Weiss" < > yishayj...@hotmail.com>: > > > > > > I think the focus should be to agree on the requirements. > > Whether an > > > RM uses the tool created by Chris or Alex can be a matter of > choice. > > > > > > From: Carlos Rovira<mailto:carlosrov...@apache.org> > > > Sent: Wednesday, March 25, 2020 5:18 PM > > > To: Apache Royale Development<mailto:dev@royale.apache.org > > > > > Subject: Re: Releasing: Finally giving up > > > > > > Hi, > > > > > > first of all, many thanks for all the time invested by > Chris this > > > days. We > > > almost didn't have any normal life this latest 2'5 days, > but as > > well > > > many > > > other work (spliced in many hours in the several previous > days) > > was > > > prepared to start this release, so for a person out side > this > > project, > > > I > > > think we all should thanks a lot his dedications. The good > > output is > > > that > > > process was streamlined, removing unneeded params to enter > and > > fixing > > > many > > > bugs over the place. That only can be performed by a good > > specialist in > > > build systems like Chris, I just could be a companion to > execute > > steps > > > and > > > try to be trained in the process. > > > > > > About me: I must to say that I tried hard to make this > work, but > > I must > > > assume I don't have the skills to do it (what is normal, > since > > if an > > > expert > > > in build stuff like Chris can't do it, I don't think any > other > > one > > > could do > > > it, and less me). The good point is that I know I have a > great a > > deep > > > knowledge about the overall process, and while in November > I > > could > > > envision, something of what I could see this days, now I > have a > > clear > > > knowledge of all of it (taking into account that we just go > > through > > > compiler and typedefs and not framework, that could be, > due to > > size, a > > > much > > > bigger challenge...) > > > > > > About the process: I must say that the initial intention > was so > > good, > > > and I > > > think we all want the premise offered thanks to Alex. The > > reality I > > > think > > > is defeating the premise, and we should all see the > reality with > > clear > > > sight and without worrying about egos but just looking and > what > > the > > > project > > > need. > > > > > > Since the CI process involves so many steps, manual > commands in > > CI > > > steps, > > > and...manual commands in local machine. One of the most > annoying > > > things is > > > the CI server hanging lots of time (the Java agent exits > and that > > > means you > > > must reboot in order the system working all again without > > problems). > > > The > > > final problem is that you as a RM will fail some times in > doing > > what > > > the > > > process needs (indicated in emails), and although you are > > trained a > > > lot in > > > how to do the project you will found stuck in some > situations > > where you > > > simple don't know what to do. > > > > > > (For example I forgot or did a bad step that makes me > rollover > > things > > > in > > > succeed steps, so I had to roll back versions in the > compiler, > > but > > > since I > > > was doing a tools release too, I didn't have into account > that I > > need > > > to > > > roll back versions in tools too. It's clear as Chris notice > > that, but > > > not > > > for me at all. So we all should know with this example > that the > > > process is > > > not automatic at all. If just involve 2-3 steps, that > would be > > > manageable, > > > but we are talking of 13 steps with 32 manual steps (until > step > > 7), > > > plus > > > maybe other 30 or so commands until step 13?)...hope you > > understand the > > > challenge here and that is nothing easy for any of us. > > > > > > What I mean, and my advice, from someone that loves this > project > > and > > > want > > > it to make it shine: We should try to simple focus in > what's > > important. > > > Chris is offering a way that ensures release done simple > and > > easily. My > > > take is that we can't afford to go this way more time, > since the > > > project > > > will see his existence threatened. We're here many years > here, > > and this > > > actual problem is a big one. It's clear that we have a > problem > > if we > > > can > > > release every month (or two month) with a process that is > easy > > for > > > everyone > > > and requires not a great training to do so, and will > require > > some few > > > commands and operations to be done. We simply can't > release, and > > we can > > > rely in a process that is so fragile, to make every change > we do > > in the > > > future will break it. > > > > > > The fact is that we didn't get so many people jumping to > Royale > > and the > > > ones with an eye on this project can finally gone if we > continue > > > trying to > > > force things that we don't need. We can release just with > maven > > in few > > > steps, that's a fact, and we don't really need to make > things in > > the > > > actual > > > way. If we do is because we're forcing it, but nobody > cares, and > > my > > > vision > > > is that will cost the project. > > > > > > We simplify the build Maven process to make it more > reliable, > > that > > > break > > > the release process in November. The build process has > priority > > over > > > the > > > release process, since lots of users depend on them (royale > > devs, and > > > users). The release process is important in the project > context > > to > > > offer > > > the bits to the public as a tag or point in time. > > > > > > We simply can't put the release process to the service of > the > > build > > > process > > > and make all of that stop in time to not break a fragile > release > > > process. > > > That's not an option for an open source project. The > release > > process > > > is at > > > the service of the rest (people, builds, code....) > > > > > > I'd want people here to rethink all of this since: > > > > > > > > > - We can't afford this project not to release easily > monthly > > (or > > > every 2 > > > months) for anyone here (and I mean all people here that > > wants to > > > be an RM). > > > - We can't afford so many hours invested to release, or > to > > maintain > > > a > > > process that is very, very, very difficult to operate, > > fragile and > > > easy to > > > break. > > > > > > > > > The motivations are: > > > > > > > > > 1. Although the process was designed and done with very > good > > > intentions, > > > the final product does not match the premise. "To make > the > > Release > > > process > > > affordable and reliable for each RM" > > > 2. The CI server steps needs you to work on you local > > machine, what > > > defeats the premise to work just in the CI server. > > > 3. The actual process involve 13 steps instead of > something > > more > > > manageable like 2-3 > > > 4. The actual process involve many manual commands in > the CI > > server > > > splitter through the 13 steps (1 to 5 commands for each > step > > > depending on > > > the step) > > > 5. The actual process involve many manual commands in > your > > local > > > machine > > > splittled every 3 steps. So you need to use your local > > machine what > > > defeats > > > entirely the CI steps to not need the RM to configure > his > > machine. > > > This > > > happens in steps 3 and 7 (and fo sure later in > frameworks > > part). > > > 6. The CI server use to hang the Jenkins java agent > process > > that > > > involves a reboot, what make all the process highly > > cumbersome. You > > > can > > > easily get to reboot CI server 1-2 times while doing the > > first 3 > > > steps. > > > 7. If you fail in some step (because you simply fail > since > > you're > > > human, > > > or the hangs and things happening make you loose the > track of > > what > > > you're > > > doing and make something that provoke the fail) you > need to > > > rollback, and > > > know how to operate to restore al the process, crossing > > fingers to > > > expect > > > all goes well, rolling back versions, removing release > branch, > > > removing > > > tags,...) > > > 8. In the end, If you finally get all the process to > work, > > since is > > > very, very fragile, is very possible that some updates > in the > > > future make > > > it fail again, and the fix will required more and more > hours > > of > > > experimented people, patience, and testing. > > > > > > About time. For people trained in the process, (and I > think I > > get very > > > trained this days). Steps 1 to 6-7 is about 1h and 30' to > 2h. I > > think > > > getting to step 13 and taking into account that framework > is > > much more > > > code > > > and time to build, I expect the global process is about 2h > or > > maybe > > > more. > > > So we are talking about 3'5 to 4h, of all this complicated > > process with > > > lots of opportunities to fail in some of the ways already > > commented. > > > > > > We have the opportunity now: Chris wanted to use the > project for > > PCL4X > > > to > > > build IoT apps. I think that's a huge opportunity for > Royale. > > He's > > > giving > > > up to use whatever other frontend tech (React, Angular, > Vue,...). > > > > > > So all this is making Royale to continue moving backwards > > instead to > > > conquer new positions. I have clear that this is not the > way to > > go, > > > and if > > > the rest of the team wants to continue forcing all of > this. I > > just can > > > do > > > anything but start thinking in other options, a fork or > > something that > > > ensure we are not loosing more for the years to come. > > > > > > I still hope all of you try to think well on all of this. > Remove > > egos, > > > since is not about you and me, is about just Apache > Royale, and > > nobody > > > out > > > there cares who or what did this or that. Is just a matter > of > > what we > > > want > > > for Royale, since people will just see what the project > can do > > for him. > > > > > > Hope this long email could make you think, and hope you > all will > > > comment to > > > have clear our positions and have clear what we can expect > in > > this > > > problematic releasing problem, since is what will make us > get > > more > > > traction > > > or in reverse, end loosing more users and maybe die in the > way. > > > > > > Thanks > > > > > > Carlos > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > El mié., 25 mar. 2020 a las 12:39, Christofer Dutz (< > > > christofer.d...@c-ware.de>) escribió: > > > > > > > Hi all, > > > > > > > > after 3-4 days of some times 10-16 hours of working on > getting > > the > > > > “process” running, I’m finally giving up. > > > > We managed to fix a lot of issues in the way the steps > were > > setup and > > > > managed to get to step 7 however there’s no getting past > that. > > > > > > > > The problem are the reproducible builds. > > > > > > > > While for the java part I think we have all bases > covered, > > however I > > > can’t > > > > see however how I can fix the compiler to actually > produce > > > reproducible > > > > builds. > > > > > > > > We are passing in the timestamp (slightly different > format, > > however > > > still > > > > correct as we also adjusted the format-string). When > building > > > locally, the > > > > timestamp is exactly as the timestamp string tells the > > compiler. On > > > the CI > > > > however there’s this 60 minute offset. I have explicitly > set > > the > > > time-zone > > > > wherever I found time-zone relevant code to UTC (Which > is the > > > general best > > > > practice for date stuff). However the offset didn’t > change. So > > I am > > > > completely out of ideas what could be causing this. If it > > really is > > > the > > > > Summertime in the US, well I don’t know how to fix it > (perhaps > > setup > > > a rule > > > > to not release in this time). > > > > > > > > Besides that, there are still differences in the library > SWF: > > > > > > > > * As I mentioned yesterday, the node typedef > contains a > > “net” > > > object > > > > definition in the local build and a “http” one on the CI > server > > > version. > > > > * In other modules it’s simply the order of items > > generated. I > > > guess > > > > it’s a similar issue as the one I fixed in the > build-tools, > > but I > > > have no > > > > idea how to fix it in the compiler > > > > > > > > So in the end I’m giving up … I think the “process” is > now way > > more > > > stable > > > > than before, but it’s still a really fragile construct. > Today > > Greg, > > > Carlos > > > > and I did it together via Zoom and still we had to re-do > some > > steps > > > because > > > > one little preparation was missed. > > > > > > > > I would strongly doubt that the reproducible builds > really > > built > > > globally > > > > reproducible builds in the past … perhaps it worked in a > narrow > > > corridor of > > > > time-zones and OS/Java versions (I did use java 1.8), > but I > > simply > > > can’t > > > > believe it ever was 100% reproducible. > > > > > > > > So as long as this reproducibility is a requirement, I > guess > > we’re > > > done. I > > > > do understand what it’s used for in this context, > however I > > think > > > it’s just > > > > not working. > > > > > > > > The other option would be to simply release with maven, > which > > would > > > have > > > > been a 1-2 hour experience for all modules. > > > > > > > > I do agree that if the steps work and all preconditions > would > > be > > > checked > > > > (which they currently are not) it is very easy to break > the > > > “process”. And > > > > it involves running 13 jobs … but as far as I got it > also the > > > execution of > > > > 32 manual commands for everything till step 7 > > > > 4 + 6 + 3 + 3 + 3 + 4 + 2 + 3 + 3 = 32 > > > > > > > > So as long as we’re forced to stick to this process … I > guess > > > someone else > > > > will have to do it. > > > > > > > > Please feel free to revert any changes I did in the last > few > > months. > > > > > > > > Signing off, > > > > Chris > > > > > > > > PS: You really should make this ApacheRoyaleCI guy a > committer > > > considering > > > > the amount of emails he’s writing ;-) > > > > > > > > > > > > > > > > > > -- > > > Carlos Rovira > > > > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Cf9c8edc66d194754b80008d7d0fc8f9d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207655475703139&sdata=t0OhZsqlucKfzx8QzyLHWIvZSFbVtHsLz238HEBvnvU%3D&reserved=0 > > > > > > > > > > > > > > > > -- > > Carlos Rovira > > > > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Cf9c8edc66d194754b80008d7d0fc8f9d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207655475703139&sdata=t0OhZsqlucKfzx8QzyLHWIvZSFbVtHsLz238HEBvnvU%3D&reserved=0 > > > > > > > > -- > Carlos Rovira > > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7Cf9c8edc66d194754b80008d7d0fc8f9d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207655475713087&sdata=IVphZFaoiw4CP3aL5JuAb0y9kzsyzQpahCODXEdWimg%3D&reserved=0 > > >
