I am not familiar with buildbot ore this sort of setup so please forgive for some simple minded questions.
Is this SW CI or HW CI or both? How does the RPi/BBB/Laptop fit into the picture. Any Pictures? David -----Original Message----- From: Fabio Balzano [mailto:fa...@elfarolab.com] Sent: Friday, December 20, 2019 5:22 AM To: dev@nuttx.apache.org Subject: Re: [DISCUSS - NuttX Workflow] 2 hours is a configured parameter, it is to allow burst of commits, it can reduced to 0 if you need real time building, then the buildbot server can also provision remote testing of the builds. > On 20 Dec 2019, at 13:09, David Sidrane <david.sidr...@nscdg.com> wrote: > > Hi Fabio, > > What are the capabilities? > > It this 1 RPi/BBB per board nuttx board? > > David > > -----Original Message----- > From: Fabio Balzano [mailto:fa...@elfarolab.com] > Sent: Friday, December 20, 2019 5:06 AM > To: dev@nuttx.apache.org > Subject: Re: [DISCUSS - NuttX Workflow] > > Hello, > > yes the buildbot server is down, later today I will bring it up. Yes you > can > do remote builds using a RPI/BBB or similars or local builds performed by > the server itself. I can setup and maintain the server for the Nuttx > project > in case you think it is useful. > > Thank you so much > Fabio Balzano > >> On 20 Dec 2019, at 13:00, Alan Carvalho de Assis <acas...@gmail.com> >> wrote: >> >> Hi David, >> >> Sorry for scolding you in public as well, but I think we don't need to >> find guilt. >> >> So, I got the impression you were doing it to promote PX4 test >> workflow as the best solution for all the NuttX issues. >> >> And although 300K drones are a lot, there are many commercial products >> using NuttX. Many Sony audio recorders, Moto Z Snaps, Thermal >> printers, etc. Probably we have products that overcome that number. >> >> I think recently Fabio changed the buildbot link. BTW I just remember >> other alternative that Sebastien and I did about 3 years ago: >> >> https://bitbucket.org/acassis/raspi-nuttx-farm/src/master/ >> >> The idea was to use low cost Raspberry PIs as a distributed build test >> for NuttX. It worked fine! You just define a board file with the >> configuration you want to test and it is done. >> >> BR, >> >> Alan >> >>> On 12/20/19, David Sidrane <davi...@apache.org> wrote: >>> Hi Alan, >>> >>> Sorry if my intent was misunderstood. I am merely stating facts on were >>> we >>> are and how got there.I am not assigning blame. I am not forcing >>> anything >>> I >>> am giving some examples of how we can make it the project complete and >>> better. We can use all of it, some of it none of it. The is a group >>> decision. >>> >>> Also Pleases do fill us in on where we can see the SW CI & HW CI you >>> mentioned. Do you have links maybe be we can use it now? >>> >>> Again Sorry! >>> >>> David >>> >>>> On 2019/12/20 11:44:23, Alan Carvalho de Assis <acas...@gmail.com> >>>> wrote: >>>> Hi David, >>>> >>>>> On 12/20/19, David Sidrane <davi...@apache.org> wrote: >>>>> Hi Nathan, >>>>> >>>>> On 2019/12/20 02:51:56, Nathan Hartman <hartman.nat...@gmail.com> >>>>> wrote: >>>>>> On Thu, Dec 19, 2019 at 6:24 PM Gregory Nutt <spudan...@gmail.com> >>>>>> wrote: >>>>>>>>> ] A bad build system change can cause serious problems for a lot >>>>>>>>> of >>>>>> people around the world. A bad change in the core OS can destroy the >>>>>> good >>>>>> reputation of the OS. >>>>>>>> Why is this the case? Users should not be using unreleased code or >>>>>>>> be >>>>>> encouraged to use it.. If they are one solution is to make more >>>>>> frequent >>>>>> releases. >>>>>>> I don't think that the number of releases is the factor. It is time >>>>>>> in >>>>>>> people's hand. Subtle corruption of OS real time behavior is not >>>>>>> easily >>>>>>> testing. You normally have to specially instrument the software >>>>>>> and >>>>>>> setup a special test environment perhaps with a logic analyzer to >>>>>>> detect >>>>>>> these errors. Errors in the core OS can persists for months and in >>>>>>> at >>>>>>> least one case I am aware of, years, until some sets up the correct >>>>>>> instrumented test. >>>>>> >>>>>> And: >>>>>> >>>>>> On Thu, Dec 19, 2019 at 4:20 PM Justin Mclean >>>>>> <jus...@classsoftware.com> >>>>>> wrote: >>>>>>>> ] A bad build system change can cause serious problems for a lot >>>>>>>> of >>>>>> people around the world. A bad change in the core OS can destroy the >>>>>> good >>>>>> reputation of the OS. >>>>>>> >>>>>>> Why is this the case? Users should not be using unreleased code or >>>>>>> be >>>>>> encouraged to use it.. If they are one solution is to make more >>>>>> frequent >>>>>> releases. >>>>>> >>>>>> Many users are only using released code. However, whatever is in >>>>>> "master" >>>>>> eventually gets released. So if problems creep in unnoticed, >>>>>> downstream >>>>>> users will be affected. It is only delayed. >>>>>> >>>>>> I can personally attest that those kinds of errors are extremely >>>>>> difficult >>>>>> to detect and trace. It does require a special setup with logic >>>>>> analyzer >>>>>> or >>>>>> oscilloscope, and sometimes other tools, not to mention a whole setup >>>>>> to >>>>>> produce the right stimuli, several pieces of software that may have >>>>>> to >>>>>> be >>>>>> written specifically for the test.... >>>>>> >>>>>> I have been wracking my brain on and off thinking about how we could >>>>>> set >>>>>> up >>>>>> an automated test system to find errors related to timing etc. >>>>>> Unfortunately unlike ordinary software for which you can write an >>>>>> automated >>>>>> test suite, this sort of embedded RTOS will need specialized hardware >>>>>> to >>>>>> conduct the tests. That's a subject for another thread and i don't >>>>>> know >>>>>> if >>>>>> now is the time, but I will post my thoughts eventually. >>>>>> >>>>>> Nathan >>>>> >>>>> From the proposal >>>>> >>>>> "Community >>>>> >>>>> NuttX has a large, active community. Communication is via a Google >>>>> group at >>>>> https://groups.google.com/forum/#!forum/nuttx where there are 395 >>>>> members as >>>>> of this writing. Code is currently maintained at Bitbucket.org at >>>>> https://bitbucket.org/nuttx/. Other communications are through >>>>> Bitbucket >>>>> issues and also via Slack for focused, interactive discussions." >>>>> >>>>> >>>>>> Many users are only using released code. >>>>> >>>>> Can we ask the 395 members? >>>>> >>>>> I can only share my experience with NuttX since I began working on the >>>>> project in 2012 for multiple companies. >>>>> >>>>> Historically (based on my time on the project) releases - were build >>>>> tested >>>>> - by this I mean that the configurations were updated and the thus >>>>> created a >>>>> set of "Build Test vectors" BTV. Given the number of permutations >>>>> solely >>>>> based on the load time of >>>>> (http://nuttx.org/doku.php?id=documentation:configvars) with 95,338 >>>>> CONFIG_* >>>>> hits. Yes there are duplicates on the page and dependencies. This is >>>>> just >>>>> meant to give a number of bits.... >>>>> >>>>> The total space is very large >>>>> >>>>> The BTV space was very sparse coverage. >>>>> >>>>> IIRC Greg gave the build testing task a day of time. It was repeated >>>>> after >>>>> errors were found. I am not aware of any other testing. Are you? >>>>> >>>>> There were no Release Candidate (rc) nor alpha nor beta test that ran >>>>> this >>>>> code one real systems and very little, if any Run Test Vectors (RTV) - >>>>> I >>>>> have never seen a test report - has anyone? >>>>> >>>>> One way to look at this is Sporadic Integration. (SI) with limited BTV >>>>> and >>>>> minimal RTV. Total Test Vector Coverage TTVC = BTV + RTV; The ROI of >>>>> way >>>>> of working, from a reliability perspective was and is very small. >>>>> >>>>> A herculean effort Greg's part with little return: We released code >>>>> with >>>>> many significant and critical errors in it. See the ReleaseNotes and >>>>> the >>>>> commit log. >>>>> >>>>> Over the years Greg referred to TRUNK (yes it was on SVN) and master >>>>> as >>>>> his >>>>> "own sandbox" stating is should not be considered stable or >>>>> build-able. >>>>> This >>>>> is evident in the commit log. >>>> >>>> Please stop focusing on the people (Greg) and let talk about how the >>>> workflow. >>>> We are here to discuss how we can improve the process, we are not >>>> talking about throw away NuttX Build System and move to PX4. >>>> >>>> You are picturing something that is not true. >>>> >>>> We have issues, as FreeRTOS, MBEB and Zephyr also have. But it is not >>>> Greg or the Build System guilt. >>>> >>>> Please, stop! It is disgusting! >>>> >>>>> I have personally never used a release from a tarball. Given the above >>>>> why >>>>> would I? It is less stable then master at TC = N >>>>> (https://www.electronics-tutorials.ws/rc/rc_1.html) where N Is some >>>>> number >>>>> of days after a release. - unfortunately based on the current >>>>> practices >>>>> (a >>>>> very unprofessional workflow) N is also dictated by when apps and >>>>> nuttx >>>>> actually building for a given target's set of BTV. >>>> >>>> It is not "unprofessional" it was what we could do based or our >>>> hardware limitations. >>>> >>>>> With the tools and resources that exist in our work today, Quite >>>>> frankly: >>>>> This unacceptable and is an embarrassment. >>>> >>>> Oh my Gosh! Please don't do it. >>>> >>>> >>>>> I suspect this is why there is a Tizen. The modern era - gets it. >>>>> (Disclaimer I am an old dog - I am learning to get it) >>>> >>>> Tizen exists because companies want to have control. >>>> This is the same logic why Redhat and others maintain their own Linux >>>> kernel by themselves. >>>> >>>>> --- Disclaimer --- >>>>> >>>>> In the following, I'm am not bragging about PX4 or selling tools, I am >>>>> merely trying to share our experiences for the betterment of NuttX. >>>>> >>>>> From what I understand PX4 has the most instances of NuttX running on >>>>> real >>>>> HW in the world. Over 300K. (I welcome other users to share their >>>>> numbers) >>>>> >>>>> PX4's Total TTVC is still limited, but much, much greater than NuttX. >>>>> >>>>> We use Continuous integration (CI) on Nuttx on PX4 on every commit on >>>>> PRs. >>>>> >>>>> C/C++ CI / build (push) Successful in 3m >>>>> Compile MacOS Pending — This commit is being built >>>>> Compile All Boards — This commit looks good >>>>> Hardware Test — This commit looks good >>>>> SITL Tests — This commit looks good >>>>> SITL Tests (code coverage) — This commit looks good >>>>> ci/circleci — Your tests passed on CircleCI! >>>>> continuous-integration/appveyor/pr — AppVeyor build succeeded >>>>> continuous-integration/jenkins/pr-head — This commit looks good >>>>> >>>>> >>>>> We run tests on HW. >>>>> >>>>> http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-mag-str-preflt/1/pipeline >>>>> >>>>> I say limited because of the set of arch we use and the way we >>>>> configure >>>>> the >>>>> OS. >>>>> >>>>> I believe this to be true of all users. >>>>> >>>>> The benefit of a community is that the sum of all TTVC that finds the >>>>> problems and fix them. >>>>> >>>>> Why not maximize TTVC - if it will have a huge ROI and it is free: >>>>> >>>>> PX4 will contribute all that we have. We just need to build temporally >>>>> consistent build. Yeah he is on the submodule thing AGAIN :) >>>> >>>> Just to make the history short: we already have solutions for SW and HW >>>> CI. >>>> >>>> Besides the buildbot (https://buildbot.net) that was implemented and >>>> tested by Fabio Balzano, Xiaomi also has a build test for NuttX. >>>> >>>> At end of the day, it is not only Greg testing the system, we all are >>>> testing it as well. >>>> >>>> Don't try to push PX4 down your throat, it will not work this way. >>>> Let's keep the Apache way, it is a democracy! >>>> >>>> BR, >>>> >>>> Alan >>>
