IMO, reproducible binaries are useful for more than building artifacts on a
remote machine. I think enough folks agree that they produced plugins for
Maven to help with reproducibility. When I introduced it to Royale, it wasn't
just to solve the CI build issue, it was to set us up for the future, in case
signed binaries become required for installation for security/anti-malware
reasons.
But as I suggested in my previous post, it might help to first come up with the
set of maven commands required to reproduce (using your local machine) the same
number of files with the same names (other than the version portion) that were
in 0.9.6.
-Alex
On 3/25/20, 12:28 PM, "Christofer Dutz" <christofer.d...@c-ware.de> wrote:
Hi Alex,
this is again under the premise that releases "must" be done on the CI
server, which I would personally like to question.
Royale is, as far as I know, the only project insisting on this.
If you build the release on your machine you don't have to check if this is
really what you build on your machine.
Chris
Am 25.03.20, 19:23 schrieb "Alex Harui" <aha...@adobe.com.INVALID>:
That has nothing to do with the Ant build of release artifacts. That
is just using Ant to run command line commands to download and verify Maven
release artifacts. I'm pretty sure you can just look at the steps and type
them in manually at the command line and you'll get the same results.
AFAICT, the problem is that the reproducible binaries generated by
Maven aren't reproducible. It could be an issue when the US and Europe have
different daylight savings time settings. When we saw it work, we were not at
one of those transition points. So maybe it is best to just wait until Sunday.
But it did work for me in the US and Piotr in EU.
Proposals for better ways to have an RM verify artifacts built on a
remote machine are welcome. Testing reproducible binaries was the only way I
could think of and implement.
-Alex
On 3/25/20, 11:13 AM, "Carlos Rovira" <carlosrov...@apache.org> wrote:
Hi Alex,
after run CI Server 007 we get this email:
From the royale-typedefs repo:
1. Run ant -f releasesteps.xml Release_Step_007
-Drelease.version=0.9.7
-DskipTests=true
This will download the artifacts then unzip and compile the source
artifact.
2. Validate that the compiled artifacts match the downloaded
artifacts.
3. If they do, then run ant -f releasesteps.xml
Release_Step_007_Sign
-Drelease.version=0.9.7
This will PGP sign the source ZIP and compiled JARs
4. Then run ant -f releasesteps.xml Release_Step_007_Upload
-Drelease.version=0.9.7
This will upload the signed artifacts to Maven Release Staging. Do
not
"Close" the staging repository until the other repos have been
added.
This is what is failing from sub step 007-1, so as part of the
step, I'm
downloading with ant and building, signing and then uploading (Or
steps
should allow me to do that, but is failing)
A part from that I don't think we really want a solution where of
breaking
steps event more. What we're requesting is remove steps.
El mié., 25 mar. 2020 a las 18:54, Alex Harui
(<aha...@adobe.com.invalid>)
escribió:
> I'm sorry, I must have missed something. Where was Ant involved
in any of
> the problems reported so far? AFAICT, it was all Maven plugins
driving our
> compiler and we couldn't get reproducible binaries.
>
> I suggested in one reply that Chris fills a staging repo using
his local
> machine just to make sure we know what Maven commands need to be
run to
> generate all of the Maven artifacts. And that would validate
the claim
> that it is really doable in two or three steps. A partial
verification of
> validity is to check that the same set of files was produced as
in 0.9.6
> (there might be some new SWCs though).
>
> Then someone else (maybe Carlos) would download all of the
artifacts, run
> the same steps, download the new artifacts and verify that the
binary
> match. Then we can work on splitting those steps into chunks for
the CI
> server.
>
> -Alex
>
>
> On 3/25/20, 10:13 AM, "Carlos Rovira" <carlosrov...@apache.org>
wrote:
>
> We imposed ourselves a need to release building with two
build systems.
> That's the biggest issue and that's a requirement of this PMC.
> That's what makes the need of the actual CI steps and the
complex and
> intricate process.
>
> For the rest of Apache a release is:
>
> "For an *Apache* project, that means any publication outside
the
> development community, defined as individuals actively
participating in
> development or following the dev list. More narrowly, an
official
> *Apache
> release* is one which has been endorsed as an "act of the
Foundation"
> by a
> PMC."
>
> That means that we can release with ANT, with Maven, or any
other build
> systems as part of the process, and even try what we're doing
of
> trying to
> build with n build systems and try to check all is ok with the
> artifacts
> generated by all that build systems and check all of
them...but if
> that's
> what we want, I personally will left this wagon, and wish you
the best
> of
> lucks in the process.
>
> Chris and I proposed to just use Maven *for release* (so
please, I
> expect
> no body starts to say we are throwing away ANT from the
project for
> building, please....we are just discussing release here).
>
> What do you think about to try to release just with Maven? If
can do
> in the
> next day or two and get the release published...will that
work for
> you? I
> think we don't need anything more, and I'm sure our users
will be
> happy too.
>
> And yes, people wanting then to build with Ant or with Maven
will be
> able
> to do so... :-)
>
> Thanks
>
> Carlos
>
>
> [1]
>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Fdev%2Frelease-publishing.html&data=02%7C01%7Caharui%40adobe.com%7C2686c84c4497421753d408d7d0f2bc4c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207613278446831&sdata=gkyY1LLl6N77RV7aUCPibF355zPc158YXa4xnXogDvk%3D&reserved=0
>
>
>
> El mié., 25 mar. 2020 a las 17:46, Christofer Dutz (<
> christofer.d...@c-ware.de>) escribió:
>
> > Hi Yiashay,
> >
> > I agree ... but I didn't create Maven ... haven't even
provided a
> single
> > PR to it ;-)
> > I think my only noticeable contribution was that they
changed their
> > website to forbid the usage of "maven-{something}-plugin"
to Apache
> Maven
> > Core modules (It initially said Apache Maven Plugins which
would have
> > applied for maven-flex-plugin and maven-plc4x-plugin)
> >
> > I think you should focus not on the process, but the bases
a process
> > should cover.
> >
> > Focus on what you want to be provided or ensured by any
process, no
> matter
> > which tool it is based on.
> >
> > That's what I created the google doc for.
> >
> > If you decide to loosen the requirement on sticking 100% to
the
> existing
> > process I am more than willing to help.
> > However if you stick to the current process, it doesn't
make any
> sense for
> > me to do so.
> >
> > Looking forward to a productive discussion,
> >
> > Chris
> >
> >
> > Am 25.03.20, 17:36 schrieb "Yishay Weiss"
<yishayj...@hotmail.com>:
> >
> > I think the focus should be to agree on the
requirements.
> Whether an
> > RM uses the tool created by Chris or Alex can be a matter
of choice.
> >
> > From: Carlos Rovira<mailto:carlosrov...@apache.org>
> > Sent: Wednesday, March 25, 2020 5:18 PM
> > To: Apache Royale
Development<mailto:dev@royale.apache.org>
> > Subject: Re: Releasing: Finally giving up
> >
> > Hi,
> >
> > first of all, many thanks for all the time invested by
Chris this
> > days. We
> > almost didn't have any normal life this latest 2'5
days, but as
> well
> > many
> > other work (spliced in many hours in the several
previous days)
> was
> > prepared to start this release, so for a person out
side this
> project,
> > I
> > think we all should thanks a lot his dedications. The
good
> output is
> > that
> > process was streamlined, removing unneeded params to
enter and
> fixing
> > many
> > bugs over the place. That only can be performed by a
good
> specialist in
> > build systems like Chris, I just could be a companion
to execute
> steps
> > and
> > try to be trained in the process.
> >
> > About me: I must to say that I tried hard to make this
work, but
> I must
> > assume I don't have the skills to do it (what is
normal, since
> if an
> > expert
> > in build stuff like Chris can't do it, I don't think
any other
> one
> > could do
> > it, and less me). The good point is that I know I have
a great a
> deep
> > knowledge about the overall process, and while in
November I
> could
> > envision, something of what I could see this days, now
I have a
> clear
> > knowledge of all of it (taking into account that we
just go
> through
> > compiler and typedefs and not framework, that could be,
due to
> size, a
> > much
> > bigger challenge...)
> >
> > About the process: I must say that the initial
intention was so
> good,
> > and I
> > think we all want the premise offered thanks to Alex.
The
> reality I
> > think
> > is defeating the premise, and we should all see the
reality with
> clear
> > sight and without worrying about egos but just looking
and what
> the
> > project
> > need.
> >
> > Since the CI process involves so many steps, manual
commands in
> CI
> > steps,
> > and...manual commands in local machine. One of the most
annoying
> > things is
> > the CI server hanging lots of time (the Java agent
exits and that
> > means you
> > must reboot in order the system working all again
without
> problems).
> > The
> > final problem is that you as a RM will fail some times
in doing
> what
> > the
> > process needs (indicated in emails), and although you
are
> trained a
> > lot in
> > how to do the project you will found stuck in some
situations
> where you
> > simple don't know what to do.
> >
> > (For example I forgot or did a bad step that makes me
rollover
> things
> > in
> > succeed steps, so I had to roll back versions in the
compiler,
> but
> > since I
> > was doing a tools release too, I didn't have into
account that I
> need
> > to
> > roll back versions in tools too. It's clear as Chris
notice
> that, but
> > not
> > for me at all. So we all should know with this example
that the
> > process is
> > not automatic at all. If just involve 2-3 steps, that
would be
> > manageable,
> > but we are talking of 13 steps with 32 manual steps
(until step
> 7),
> > plus
> > maybe other 30 or so commands until step 13?)...hope you
> understand the
> > challenge here and that is nothing easy for any of us.
> >
> > What I mean, and my advice, from someone that loves
this project
> and
> > want
> > it to make it shine: We should try to simple focus in
what's
> important.
> > Chris is offering a way that ensures release done
simple and
> easily. My
> > take is that we can't afford to go this way more time,
since the
> > project
> > will see his existence threatened. We're here many
years here,
> and this
> > actual problem is a big one. It's clear that we have a
problem
> if we
> > can
> > release every month (or two month) with a process that
is easy
> for
> > everyone
> > and requires not a great training to do so, and will
require
> some few
> > commands and operations to be done. We simply can't
release, and
> we can
> > rely in a process that is so fragile, to make every
change we do
> in the
> > future will break it.
> >
> > The fact is that we didn't get so many people jumping
to Royale
> and the
> > ones with an eye on this project can finally gone if we
continue
> > trying to
> > force things that we don't need. We can release just
with maven
> in few
> > steps, that's a fact, and we don't really need to make
things in
> the
> > actual
> > way. If we do is because we're forcing it, but nobody
cares, and
> my
> > vision
> > is that will cost the project.
> >
> > We simplify the build Maven process to make it more
reliable,
> that
> > break
> > the release process in November. The build process has
priority
> over
> > the
> > release process, since lots of users depend on them
(royale
> devs, and
> > users). The release process is important in the project
context
> to
> > offer
> > the bits to the public as a tag or point in time.
> >
> > We simply can't put the release process to the service
of the
> build
> > process
> > and make all of that stop in time to not break a
fragile release
> > process.
> > That's not an option for an open source project. The
release
> process
> > is at
> > the service of the rest (people, builds, code....)
> >
> > I'd want people here to rethink all of this since:
> >
> >
> > - We can't afford this project not to release easily
monthly
> (or
> > every 2
> > months) for anyone here (and I mean all people here
that
> wants to
> > be an RM).
> > - We can't afford so many hours invested to release,
or to
> maintain
> > a
> > process that is very, very, very difficult to
operate,
> fragile and
> > easy to
> > break.
> >
> >
> > The motivations are:
> >
> >
> > 1. Although the process was designed and done with
very good
> > intentions,
> > the final product does not match the premise. "To
make the
> Release
> > process
> > affordable and reliable for each RM"
> > 2. The CI server steps needs you to work on you local
> machine, what
> > defeats the premise to work just in the CI server.
> > 3. The actual process involve 13 steps instead of
something
> more
> > manageable like 2-3
> > 4. The actual process involve many manual commands
in the CI
> server
> > splitter through the 13 steps (1 to 5 commands for
each step
> > depending on
> > the step)
> > 5. The actual process involve many manual commands
in your
> local
> > machine
> > splittled every 3 steps. So you need to use your
local
> machine what
> > defeats
> > entirely the CI steps to not need the RM to
configure his
> machine.
> > This
> > happens in steps 3 and 7 (and fo sure later in
frameworks
> part).
> > 6. The CI server use to hang the Jenkins java agent
process
> that
> > involves a reboot, what make all the process highly
> cumbersome. You
> > can
> > easily get to reboot CI server 1-2 times while doing
the
> first 3
> > steps.
> > 7. If you fail in some step (because you simply fail
since
> you're
> > human,
> > or the hangs and things happening make you loose the
track of
> what
> > you're
> > doing and make something that provoke the fail) you
need to
> > rollback, and
> > know how to operate to restore al the process,
crossing
> fingers to
> > expect
> > all goes well, rolling back versions, removing
release branch,
> > removing
> > tags,...)
> > 8. In the end, If you finally get all the process to
work,
> since is
> > very, very fragile, is very possible that some
updates in the
> > future make
> > it fail again, and the fix will required more and
more hours
> of
> > experimented people, patience, and testing.
> >
> > About time. For people trained in the process, (and I
think I
> get very
> > trained this days). Steps 1 to 6-7 is about 1h and 30'
to 2h. I
> think
> > getting to step 13 and taking into account that
framework is
> much more
> > code
> > and time to build, I expect the global process is about
2h or
> maybe
> > more.
> > So we are talking about 3'5 to 4h, of all this
complicated
> process with
> > lots of opportunities to fail in some of the ways
already
> commented.
> >
> > We have the opportunity now: Chris wanted to use the
project for
> PCL4X
> > to
> > build IoT apps. I think that's a huge opportunity for
Royale.
> He's
> > giving
> > up to use whatever other frontend tech (React, Angular,
Vue,...).
> >
> > So all this is making Royale to continue moving
backwards
> instead to
> > conquer new positions. I have clear that this is not
the way to
> go,
> > and if
> > the rest of the team wants to continue forcing all of
this. I
> just can
> > do
> > anything but start thinking in other options, a fork or
> something that
> > ensure we are not loosing more for the years to come.
> >
> > I still hope all of you try to think well on all of
this. Remove
> egos,
> > since is not about you and me, is about just Apache
Royale, and
> nobody
> > out
> > there cares who or what did this or that. Is just a
matter of
> what we
> > want
> > for Royale, since people will just see what the project
can do
> for him.
> >
> > Hope this long email could make you think, and hope you
all will
> > comment to
> > have clear our positions and have clear what we can
expect in
> this
> > problematic releasing problem, since is what will make
us get
> more
> > traction
> > or in reverse, end loosing more users and maybe die in
the way.
> >
> > Thanks
> >
> > Carlos
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > El mié., 25 mar. 2020 a las 12:39, Christofer Dutz (<
> > christofer.d...@c-ware.de>) escribió:
> >
> > > Hi all,
> > >
> > > after 3-4 days of some times 10-16 hours of working
on getting
> the
> > > “process” running, I’m finally giving up.
> > > We managed to fix a lot of issues in the way the
steps were
> setup and
> > > managed to get to step 7 however there’s no getting
past that.
> > >
> > > The problem are the reproducible builds.
> > >
> > > While for the java part I think we have all bases
covered,
> however I
> > can’t
> > > see however how I can fix the compiler to actually
produce
> > reproducible
> > > builds.
> > >
> > > We are passing in the timestamp (slightly different
format,
> however
> > still
> > > correct as we also adjusted the format-string). When
building
> > locally, the
> > > timestamp is exactly as the timestamp string tells the
> compiler. On
> > the CI
> > > however there’s this 60 minute offset. I have
explicitly set
> the
> > time-zone
> > > wherever I found time-zone relevant code to UTC
(Which is the
> > general best
> > > practice for date stuff). However the offset didn’t
change. So
> I am
> > > completely out of ideas what could be causing this.
If it
> really is
> > the
> > > Summertime in the US, well I don’t know how to fix it
(perhaps
> setup
> > a rule
> > > to not release in this time).
> > >
> > > Besides that, there are still differences in the
library SWF:
> > >
> > > * As I mentioned yesterday, the node typedef
contains a
> “net”
> > object
> > > definition in the local build and a “http” one on the
CI server
> > version.
> > > * In other modules it’s simply the order of items
> generated. I
> > guess
> > > it’s a similar issue as the one I fixed in the
build-tools,
> but I
> > have no
> > > idea how to fix it in the compiler
> > >
> > > So in the end I’m giving up … I think the “process”
is now way
> more
> > stable
> > > than before, but it’s still a really fragile
construct. Today
> Greg,
> > Carlos
> > > and I did it together via Zoom and still we had to
re-do some
> steps
> > because
> > > one little preparation was missed.
> > >
> > > I would strongly doubt that the reproducible builds
really
> built
> > globally
> > > reproducible builds in the past … perhaps it worked
in a narrow
> > corridor of
> > > time-zones and OS/Java versions (I did use java 1.8),
but I
> simply
> > can’t
> > > believe it ever was 100% reproducible.
> > >
> > > So as long as this reproducibility is a requirement,
I guess
> we’re
> > done. I
> > > do understand what it’s used for in this context,
however I
> think
> > it’s just
> > > not working.
> > >
> > > The other option would be to simply release with
maven, which
> would
> > have
> > > been a 1-2 hour experience for all modules.
> > >
> > > I do agree that if the steps work and all
preconditions would
> be
> > checked
> > > (which they currently are not) it is very easy to
break the
> > “process”. And
> > > it involves running 13 jobs … but as far as I got it
also the
> > execution of
> > > 32 manual commands for everything till step 7
> > > 4 + 6 + 3 + 3 + 3 + 4 + 2 + 3 + 3 = 32
> > >
> > > So as long as we’re forced to stick to this process …
I guess
> > someone else
> > > will have to do it.
> > >
> > > Please feel free to revert any changes I did in the
last few
> months.
> > >
> > > Signing off,
> > > Chris
> > >
> > > PS: You really should make this ApacheRoyaleCI guy a
committer
> > considering
> > > the amount of emails he’s writing ;-)
> > >
> > >
> > >
> >
> > --
> > Carlos Rovira
> >
>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C2686c84c4497421753d408d7d0f2bc4c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207613278446831&sdata=NJkoMDYFaGyS0R69%2BKwrgSCJ44Lo0c34g9u2eOsizHs%3D&reserved=0
> >
> >
> >
> >
>
> --
> Carlos Rovira
>
>
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C2686c84c4497421753d408d7d0f2bc4c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207613278446831&sdata=NJkoMDYFaGyS0R69%2BKwrgSCJ44Lo0c34g9u2eOsizHs%3D&reserved=0
>
>
>
--
Carlos Rovira
https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fabout.me%2Fcarlosrovira&data=02%7C01%7Caharui%40adobe.com%7C2686c84c4497421753d408d7d0f2bc4c%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637207613278446831&sdata=NJkoMDYFaGyS0R69%2BKwrgSCJ44Lo0c34g9u2eOsizHs%3D&reserved=0
