We could achieve the same by creating a new agent (separate vm) and attaching it as a slave to the current Jenkins server.
Then we can use labels/tags to tie release jobs to one agent and non release jobs to another agent. That way we can maintain everything in one UI, one login while achieving the concurrent building capabilities we want. Jenkins is built for this kind of stuff and I think it would be best we followed this practice. Thanks, Om On Sat, Apr 18, 2020, 8:18 AM Alex Harui <[email protected]> wrote: > Om, I thought you were going to get the image from my VM and run it on > another VM. Then we'd remove all non-release jobs from your VM and all of > the release jobs from my VM. That won't solve the problem of how long it > takes for a commit to be built and tested, but will allow the RM to get a > release out without getting stuck behind the commit jobs. Is that > currently the plan? > > -Alex > > On 4/18/20, 12:43 AM, "OmPrakash Muppirala" <[email protected]> wrote: > > Makes sense to me. I will set parallel executors back to 1. > > And start working on a new Jenkins instance. > > Thanks, > Om > > > On Sat, Apr 18, 2020, 12:28 AM Alex Harui <[email protected]> > wrote: > > > Are you saying that Java, Ant and Maven do not know how to > distribute work > > to different cores? > > > > To repeat my main concern: our Jenkins jobs often use shared > resources. > > If you want to take the time to fix that, feel free, but I do not > know how > > you are going to keep two jobs from both trying to launch Flash > Player > > Debugger at the same time and open the same socket to communicate > with it, > > or try to parse the flashlog.txt that they have both written to. > > > > The royale-asjs job will use the royale-compiler workspace to get the > > compiler. If the compiler is building, the jars are not there and > the > > royale-asjs build fails. > > > > There is also only so much memory and some of our jobs use all of it > and > > I'm not sure if jobs slowed down due to paging memory and > writing/reading > > from disk, even if it is an SSD. > > > > I'm done for tonight, but that's what I was seeing last time I tried > to > > have both master and an agent running jobs in parallel. > > > > Thanks, > > -Alex > > > > On 4/17/20, 11:52 PM, "OmPrakash Muppirala" <[email protected]> > wrote: > > > > A typical setup would be to have 0 executors on master and at > least n > > number of executors on each agent, where n = number of cpu cores. > > I see that the current VM has 2 cores. So, I started with > setting the > > num > > executors on the current agent to 2. > > > > This should improve things quite a bit. If this doesn't work, I > will > > create a new VM and make it the slave agent of the current > Jenkins > > master. > > > > > > Thanks, > > Om > > > > On Fri, Apr 17, 2020 at 11:31 PM Alex Harui > <[email protected]> > > wrote: > > > > > On your VM or mine? I don't think mine can handle it. I used > to > > run jobs > > > of master and agent and we got lots of failures and jobs took > > forever. If > > > two jobs are trying to run tests with the shared Flash Player > > Debugger, for > > > example, you won't get consistent results. > > > > > > -Alex > > > > > > On 4/17/20, 11:23 PM, "OmPrakash Muppirala" < > [email protected]> > > wrote: > > > > > > I just realized that we have only one Agent (node) listed > under > > > Jenkins. I > > > am going to add at least one more node to start with. > > > > > > Thanks, > > > Om > > > > > > On Thu, Apr 16, 2020 at 9:58 AM Alex Harui > > <[email protected]> > > > wrote: > > > > > > > I think I followed the instructions and enabled SSH. > > > > > > > > On 4/16/20, 9:37 AM, "Yishay Weiss" < > [email protected]> > > wrote: > > > > > > > > I haven’t gotten it to work. Could be ssh needs to be > > enabled on > > > the > > > > VM [1]. This is nice to have but not essential. So Alex, > let > > me know > > > if you > > > > want to try that, otherwise I’ll suspend my efforts on > this. > > > > > > > > [1] > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftechinpieces%2Fpractical-azure-how-to-enable-ssh-on-azure-vm-84d8fba8103e&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=%2FdPPdqLaeWZYyshv14%2BhhL0ffSznS%2BAUOGQ3xNnwido%3D&reserved=0 > > > > > > > > From: Yishay Weiss<mailto:[email protected]> > > > > Sent: Wednesday, April 15, 2020 5:09 PM > > > > To: [email protected]<mailto: > [email protected]> > > > > Subject: RE: Prioritize Release Jobs on CI > > > > > > > > I’d like to try and install an ssh server. Can anyone > > think of a > > > > reason not to do that? I’m asking in terms of security, > and if > > there > > > might > > > > be a problem because it’s a VM. > > > > > > > > Thanks. > > > > > > > > From: Yishay Weiss<mailto:[email protected]> > > > > Sent: Monday, April 13, 2020 10:02 AM > > > > To: [email protected]<mailto: > [email protected]> > > > > Subject: RE: Prioritize Release Jobs on CI > > > > > > > > Related, as most (all?) of the instructions don’t > require > > > anything but > > > > a shell wouldn’t it be easier and faster to access the > machine > > with > > > ssh > > > > [1], rather than using remote desktop? > > > > > > > > [1] > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fwindows-server%2Fadministration%2Fopenssh%2Fopenssh_install_firstuse&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=pKNVfpAa0zGTTCz2jnsF9Uley6b%2BY%2Bg1ZlSWJ0%2FiYcY%3D&reserved=0 > > > > > > > > From: Yishay Weiss<mailto:[email protected]> > > > > Sent: Monday, April 13, 2020 9:55 AM > > > > To: [email protected]<mailto: > [email protected]> > > > > Subject: RE: Prioritize Release Jobs on CI > > > > > > > > Can’t you do that with remote desktop? > > > > > > > > ________________________________ > > > > From: OmPrakash Muppirala <[email protected]> > > > > Sent: Monday, April 13, 2020 9:51:25 AM > > > > To: Apache Royale Development <[email protected] > > > > > > Subject: Re: Prioritize Release Jobs on CI > > > > > > > > Alex, > > > > > > > > I believe this is your personal Azure subscription > right? > > Any > > > chance > > > > I can > > > > get access to it so I can try to copy the image out > the > > machine > > > and > > > > redeploy it somewhere else? > > > > > > > > Thanks, > > > > Om > > > > > > > > On Sun, Apr 12, 2020 at 6:18 PM OmPrakash Muppirala < > > > > [email protected]> > > > > wrote: > > > > > > > > > I can help set this up on Azure. Give me some > time to > > work > > > out the > > > > > details? > > > > > > > > > > Thanks, > > > > > Om > > > > > > > > > > On Sun, Apr 12, 2020, 11:49 AM Harbs < > > [email protected]> > > > wrote: > > > > > > > > > >> OK. Good to know. > > > > >> > > > > >> > On Apr 12, 2020, at 9:45 PM, Alex Harui > > > <[email protected] > > > > > > > > > >> wrote: > > > > >> > > > > > >> > FWIW, I just looked and the longest job in the > release > > > steps so > > > > far is > > > > >> 8 minutes. Most are under 2 minutes. There > might be > > jobs > > > later > > > > that take > > > > >> longer that we haven't run yet. IMO, the issue > isn't > > speed > > > of the > > > > machine, > > > > >> it is just that we are sharing the machine with > longer > > jobs > > > (1 hour > > > > for > > > > >> TourDeFlexMigration). And again, the machine > will be > > idle for > > > > stretches of > > > > >> time while the RM verifies artifacts after each > step. > > > > >> > > > > > >> > -Alex > > > > >> > > > > > >> > On 4/12/20, 11:32 AM, "Harbs" < > [email protected] > > > <mailto: > > > > >> [email protected]>> wrote: > > > > >> > > > > > >> > Fair enough. > > > > >> > > > > > >> > I expect builds to be somewhere between 10 > and 20 > > times > > > faster > > > > on a > > > > >> powerful machine. > > > > >> > > > > > >> > Yeah. It’s probably going to be a bit of work > > changing > > > the > > > > server, > > > > >> but probably worth it in the long run. > > > > >> > > > > > >> > I think I’ll try this when I do the next > release > > unless > > > Yishay > > > > wants > > > > >> to work with me on this for this release — but > I’m not > > going > > > to be > > > > able to > > > > >> help until after Passover (i.e. next week). > > > > >> > > > > > >> > Thanks, > > > > >> > Harbs > > > > >> > > > > > >> >> On Apr 12, 2020, at 9:22 PM, Alex Harui > > > <[email protected] > > > > > > > > > >> wrote: > > > > >> >> > > > > >> >> The Azure portal says: Standard F2s_v2 (2 > vcpus, 4 > > GiB > > > memory) > > > > >> >> > > > > >> >> I think I am reading changes to the build > process in > > your > > > > >> suggestions. I do not really want to spend more > of my > > time > > > on this > > > > >> process. But if you want to do the work, that's > fine > > with me. > > > > >> >> > > > > >> >> -Alex > > > > >> >> > > > > >> >> On 4/12/20, 10:57 AM, "Harbs" < > [email protected] > > > <mailto: > > > > >> [email protected]> <mailto: > [email protected] > > <mailto: > > > > >> [email protected]>>> wrote: > > > > >> >> > > > > >> >> What kind of horsepower is included in the > free > > Azure > > > account? > > > > >> >> > > > > >> >> The server I mentioned builds (considerably) > > faster than > > > my own > > > > >> local machine. The ci server seems to build many > times > > slower. > > > > >> >> > > > > >> >> One thing we can do to minimize running > server time > > > would be to > > > > >> transfer the artifacts to storage instead of > keeping > > them on > > > the > > > > server. On > > > > >> AWS, I’d probably use S3. Not sure what the > similar > > service on > > > > Azure is > > > > >> called. > > > > >> >> > > > > >> >>> On Apr 12, 2020, at 8:26 PM, Alex Harui > > > > <[email protected] > > > > >> <mailto:[email protected]>> wrote: > > > > >> >>> > > > > >> >>> OK, that's pretty much how I understand Azure > as > > well. > > > The key > > > > thing > > > > >> is that "running" includes time where the CI > server is > > not > > > running > > > > any > > > > >> Jenkins jobs. The CI Server steps might take > only a few > > > hours of > > > > actual > > > > >> server time, but there is time where the RM is > verifying > > > artifacts > > > > locally > > > > >> so you'd be paying for that or the RM would have > to keep > > > shutting > > > > down and > > > > >> restarting. > > > > >> >>> > > > > >> >>> Seems like it would be cheaper/simpler to get > the > > free > > > MSDN > > > > account > > > > >> and leave it running. > > > > >> >>> > > > > >> >>> -Alex > > > > >> >>> > > > > >> >>> On 4/12/20, 10:15 AM, "Harbs" < > > [email protected] > > > <mailto: > > > > >> [email protected]>> wrote: > > > > >> >>> > > > > >> >>> My experience is with AWS. > > > > >> >>> > > > > >> >>> I assume Microsoft has similar offerings, > but I > > don’t > > > have > > > > >> experience with Azure. > > > > >> >>> > > > > >> >>> AWS has on-demand EC2 instances which you > pay for > > only > > > the > > > > actual > > > > >> time that they are running.[1] > > > > >> >>> > > > > >> >>> Instances can be started and stopped via > command > > line > > > (or via > > > > the > > > > >> web interface) as long as you have valid > credentials to > > do so. > > > > >> >>> > > > > >> >>> For example: an m5.4xlarge instance has 16 > cores > > and > > > costs > > > > about > > > > >> $1.5 per hour. On a machine like that, a full > build > > would > > > probably > > > > take > > > > >> less than 10 minutes. It’s probably possible to > do a > > full > > > release > > > > with only > > > > >> a few hours of server time. > > > > >> >>> > > > > >> >>> Leaving a server like that running all the > time > > would get > > > > expensive, > > > > >> but if it’s just spun up for releases, you’d get > very > > fast > > > builds > > > > at a > > > > >> reasonable price. > > > > >> >>> > > > > >> >>> I’d be happy to pay $10-$50 (and possibly > more) per > > > release to > > > > make > > > > >> the release process painless for the RM. > > > > >> >>> > > > > >> >>> [1] > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 > > > > >> < > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 > > > > >> >< > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 > > > > >> < > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 > > > > >> >>< > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926092990578&sdata=AlpLmsJYbBpN%2BLTXIwn2ESklg5aU04KGAfWe2vhJxyc%3D&reserved=0 > > > > >> < > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 > > > > >> >< > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 > > > > >> < > > > > >> > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Faws.amazon.com%2Fec2%2Fpricing%2Fon-demand%2F&data=02%7C01%7Caharui%40adobe.com%7C15fe635dba674d69308108d7e36c2ea2%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637227926093000574&sdata=YZE52sCQqRXb6CAMtU4DW3RKXjg%2FffHhiGy5b0Yz2I0%3D&reserved=0 > > > > >> >>> > > > > >> >>> > > > > >> >>>> On Apr 12, 2020, at 7:45 PM, Alex Harui > > > > <[email protected]> > > > > >> wrote: > > > > >> >>>> > > > > >> >>>> I'm not very experienced with spinning up > > servers. The > > > CI > > > > server we > > > > >> are using is effectively free, based on a generous > > donation > > > from > > > > Microsoft > > > > >> of MSDN accounts to ASF committers. So I leave > it up > > 24/7, > > > and > > > > share the > > > > >> RDP access on private@. I think any other ASF > > committer > > > could do > > > > the > > > > >> same. IIRC, if that server actually is stopped, > I have > > to > > > use my > > > > personal > > > > >> (unshared) MSDN credentials to start it again. > AIUI, > > if I > > > > actually paid > > > > >> for the server, it would cost me to leave it > running > > even if > > > it > > > > didn't run > > > > >> jobs between releases. > > > > >> >>>> > > > > >> >>>> Is that what you are basically saying? I > think it > > might > > > be > > > > best if > > > > >> another committer got a CI server going via the MS > > donation > > > and > > > > could leave > > > > >> it up 24/7. > > > > >> >>>> > > > > >> >>>> -Alex > > > > >> >>>> > > > > >> >>>> On 4/12/20, 9:28 AM, "Harbs" < > > [email protected]> > > > wrote: > > > > >> >>>> > > > > >> >>>> I’m willing to do this. > > > > >> >>>> > > > > >> >>>> Considering that the release will be run > > infrequently, it > > > > should be > > > > >> doable to have a relatively powerful server that > could > > be > > > spun up on > > > > >> demand. This is something I have setup for my own > > releases. > > > > >> >>>> > > > > >> >>>> The only complication would be that each RM > would > > need > > > valid > > > > >> credentials to spin up the server. > > > > >> >>>> > > > > >> >>>> Harbs > > > > >> >>>> > > > > >> >>>>> On Apr 12, 2020, at 7:10 PM, Alex Harui > > > > <[email protected]> > > > > >> wrote: > > > > >> >>>>> > > > > >> >>>>> A better solution, IMO, is for someone else > to > > offer up > > > a CI > > > > server > > > > >> only for release jobs. > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
