On Mon, Jun 12, 2017 at 08:47PM, Olaf Flebbe wrote: > Hi Cos, > > The deployment CI job of bigtop-trunk use the repo and packages of the last > bigtop-release. So any change in package (i.e. fixes) will not be reflected > in the provisioner jobs. I am really tired in arguing against this.
Ah, I am not crazy then! Thanks for confirming my sanity, Olaf! So, the next question is how we can combine the night package build with Evans' deployment job. Yes, I know - the provisioning matrix server a different purpose to polish our Puppet code. But, and that's a big but - while polishing Puppet we might find out that it is a package that is problematic (BIGTOP-2165 is a case in point). In wish case we need to test the latest packages. I wonder if we can collect all packages together and build a repo on a nightly basis? Or perhaps we can directly use Jenkins' artifacts? I did the former for another project a while back and it is a-ok if you have just a handful of packages. In our case we are talking about Gigabytes of stuff being copied between the slaves, and I don't think it would be tht efficient. May be we can use a shared location to store the binary artifacts? Thoughts? Cos > > Am 12.06.2017 um 20:40 schrieb Konstantin Boudnik <[email protected]>: > > > > So, I got BIGTOP-2165 in and it has addressed the ignite service startup > > issue > > in my local testing, but I still see failing build in our CI [1] > > > > Evans, looks like I need your help to sort it our. Here's how I deploy the > > cluster: > > - first I provision hdfs/yarn/mapreduce components from our 1.2 repo > > - then I change the repo URL to my local one, where I have a freshly built > > ignite packages, and run 'puppet apply' manually > > - the service comes up just fine and I see the process running. I also can > > stop it using 'service ignite-hadoop stop' > > > > I was looking into the deployment matrix's config, but had a bit of a hard > > time deciphering where the packages are coming from. Is there a chance of a > > race-condition where the deployment might be using the packages from the > > last > > build or something or a sort? I guess the easiest answer to this would be to > > wait until the next provisioner cycle and see if the situation with CentOS' > > Ignite deployment has improved. > > > > This bug was driving me nuts for the past two years, and now that I thought > > I > > have fixed it there are two different outcomes in different environments. So > > please accept my apologies if I am barking at the wrong tree ;) > > > > Thanks for your help! > > > > [1] > > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/BUILD_SLAVE=docker-slave-02,COMPONENTS=ignite_hadoop,OS=centos6/54/console > > > > Cos > > > > On Thu, Jun 08, 2017 at 10:03AM, Konstantin Boudnik wrote: > >> Cool! I am trying to get some Ignite's dev@ help with BIGTOP-2108 - > >> hopefully, it won't be long before it is fixed. Thanks! > >> -- > >> Take care, > >> Konstantin (Cos) Boudnik > >> 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 > >> > >> Disclaimer: Opinions expressed in this email are those of the author, > >> and do not necessarily represent the views of any company the author > >> might be affiliated with at the moment of writing. > >> > >> > >> On Thu, Jun 8, 2017 at 8:55 AM, Evans Ye <[email protected]> wrote: > >>> OK. I agreed with you. > >>> Considered that most of us are making contributions in free time. > >>> Let's make the release less stressed but more solid. > >>> > >>> I plan to make 1.2.1 release at the end of the June. > >>> That should make plenty of time for blockers to be removed. > >>> > >>> Finally, I'd like to thank you all for taking actions to our deployment > >>> issues. > >>> > >>> > >>> 2017-06-08 1:11 GMT+08:00 Konstantin Boudnik <[email protected]>: > >>> > >>>> On Thu, Jun 08, 2017 at 12:57AM, Evans Ye wrote: > >>>>> Cos, > >>>>> Thanks a lot! > >>>>> > >>>>> Olaf, > >>>>> My original thought was to remove those puppet code which no one has > >>>>> interest to fix it. > >>>>> But you're right. Some of them are broken at package level... > >>>>> > >>>>> I've dig a bit in and found some of them are failing due to Java not > >>>>> presented before daemons starting up. > >>>>> I've uploaded a patch in BIGTOP-2775, which should fix the problem. > >>>>> But we still got many others failing for various reasons. > >>>>> > >>>>> Roman, > >>>>> Good idea. We can track them via JIRAs. > >>>>> > >>>>> All, > >>>>> I still hope that 1.2.1 can be released before DataWorks Summit. > >>>>> However, at this point I'd like to get everybody's thoughts that whether > >>>> we > >>>>> should release it and mark these stuffs as known issues, or we get them > >>>>> fixed and then release a fully functional release. > >>>>> Which one you're in favor? > >>>> > >>>> Eitherway is good for me. Making the release before the mid of next week > >>>> is > >>>> only possible if we have everything in place right now and only need to > >>>> do > >>>> the > >>>> release itself and vote on it. Otherwise, the timeframe is too tight... > >>>> > >>>> Cos > >>>> > >>>>> 2017-06-07 6:15 GMT+08:00 Roman Shaposhnik <[email protected]>: > >>>>> > >>>>>> On Sun, Jun 4, 2017 at 9:32 PM, Evans Ye <[email protected]> wrote: > >>>>>>> Hi Bigtopers, > >>>>>>> > >>>>>>> I've shared this on my Apache Big Data talk, but haven't got a > >>>> chance to > >>>>>>> share with you yet. > >>>>>>> This is a Jenkins matrix job to show the healthy status of our > >>>> component > >>>>>>> deployment code. > >>>>>>> Each run deploys hdfs + Y on X, where X is the OS and Y is the > >>>> component > >>>>>>> shown on the matrix: > >>>>>>> > >>>>>>> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop- > >>>>>> trunk-deployments/45/ > >>>>>> > >>>>>> This is super awesome! Thanks for spending time on putting it together! > >>>>>> > >>>>>>> Generally speaking, the status of our deployment code needs a lot of > >>>>>> effort > >>>>>>> to polish. > >>>>>>> So, for 1.2.1 release, I think we should make some changes: > >>>>>>> > >>>>>>> 1). Remove unmaintained failing components > >>>>>>> 2). Get maintained components fixed > >>>>>>> > >>>>>>> 1) might be easier to achieve, while 2) takes time. > >>>>>>> In order to maintain a good code quality, I suggest proceed with 1). > >>>>>>> > >>>>>>> Any different thoughts? > >>>>>> > >>>>>> Shall we at least start with filing JIRAs for everything that's > >>>>>> failing -- we can discuss > >>>>>> the particulars on the JIRAs then. > >>>>>> > >>>>>> I can try filing some tomorrow if nobody beats me to it (please do ;-)) > >>>>>> > >>>>>> Thanks, > >>>>>> Roman. > >>>>>> > >>>> >
