Oh, I think I have something - it happens on mysql, but not on sqlite. This is important :)
I'll dive into it. Alex On Wed, Jul 9, 2014 at 12:54 PM, Damian, Alexandru < [email protected]> wrote: > Hi Craig - I'd like to have a chat with you, are you on IRC ? > > Unfortunately, I am unable to reproduce the errors you're seeing. > I created the database at commit: > *a5531a2b8983318b99c119a87b78a9**2cf84160b8* > > and started the bitbake server and toasterui at commit: > *5306aaab07a7a9b3b0c7e8008dabbd**2e89daec28* > > and the build gets recorded without error. > > I'm thinking you have *.pyc files remaining on disk between git checkouts, > python not picking up changes, and getting old code running - but it may be > any cause. > > I'd like to debug this live with you online, do you have time for that ? > > Alex > > > > > > > On Tue, Jul 8, 2014 at 3:40 PM, Neth, Craig Allen <[email protected]> > wrote: > >> Hi Alex, >> >> Ok, thanks. Bug 6512 was filed. Do you have any speculation about >> when the problem might have been introduced? I don’t have any particular >> need to run the bleeding edge toaster server; if there is an earlier branch >> that might work I’m happy to switch branches on the server end of things. >> >> The weird thing is that I tried reverting the server back to the Daisy >> branch but that also failed with a different set of problems, I received >> errors when I ran the ‘migrate orm’ step. I figured that I had just fat >> fingered some of the setup commands but I can’t seem to find the problem. >> >> Craig >> >> From: <Damian>, Alexandru <[email protected]> >> Date: Tuesday, July 8, 2014 at 7:09 AM >> To: "Neth, Craig A" <[email protected]> >> Cc: "Lerner, Dave" <[email protected]>, "[email protected]" >> <[email protected]>, IonutX Chisanovici < >> [email protected]> >> >> Subject: Re: [Toaster] Need help with Production toaster setup >> >> Hi, >> >> Thanks for the info - >> >> No, the setup seems as it is designed to be. >> >> I guess somewhere in origin/master things broke. We don't test this setup >> on our suite, but maybe we should start testing this, thus I CC'ed Ionut. >> >> Meanwhile, I will start looking into fixing this. Can you please open a >> bug in Bugzilla so I can track it ? >> >> Cheers, >> Alex >> >> >> >> On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <[email protected] >> > wrote: >> >>> Hi Alex, >>> >>> Thanks for the response! >>> >>> I just did a git pull on the master, and then did the following steps on >>> the master machine: >>> >>> mysql> drop database toaster; >>> mysql> create database toaster; >>> >>> python bitbake/lib/toaster/manage.py syncdb >>> python bitbake/lib/toaster/manage.py migrate orm >>> python bitbake/lib/toaster/manage.py migrate bldcontrol >>> >>> (The wiki page does not mention this last one but the ‘syncdb' command >>> mentions that this one is also 'not synced’.) >>> >>> To start the master, I have written a short upstart script that does the >>> following: >>> >>> DATABASE_URL='mysql://user:pass@node:3306/toaster' python >>> $BASEDIR/bitbake/lib/toaster/manage.py runserver >>> >>> The server is up and running (I see the "This is Toaster" page). >>> >>> On the clients, I normally use a modified version of the ‘toaster’ script >>> (I called it toaster-remote) that avoids starting the local webserver but >>> doing the commands listed on the wiki page doesn't change the outcome, I >>> still see the same Assertion errors I mentioned in my initial message. >>> >>> As to your questions: We are indeed using mysql on the master. Git >>> commit ids: >>> >>> master:a5531a2b8983318b99c119a87b78a92cf84160b8 >>> client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28 >>> >>> I mentioned the commands I used above. >>> >>> Thanks for any help. As I mentioned, we did have all this working >>> previously before I updated the master, so I don’t think my fundamental >>> setup is wrong... >>> >>> Craig >>> >>> >>> >>> >>> From: <Damian>, Alexandru <[email protected]> >>> Date: Friday, July 4, 2014 at 8:06 AM >>> To: "Lerner, Dave" <[email protected]> >>> Cc: "Neth, Craig A" <[email protected]>, >>> "[email protected]" <[email protected]> >>> Subject: Re: [Toaster] Need help with Production toaster setup >>> >>> >>> >Hi Craig, >>> > >>> > >>> >The system should work with the server running master, and the clients >>> >running daisy. The production database MUST be set up with the master, >>> >and the clients MUST NOT be started with the toaster script - the script >>> >will get confused as to the current revision of the database. The flow >>> is: >>> > >>> > >>> >- start toaster on the server, and let it create / update the database >>> > >>> >- on the clients, you need to just start the bitbake server and the >>> >toasterui client before issuing command to the bitbake server - here >>> >there is the race condition that Dave already mentioned >>> > >>> > >>> > >>> >If you need to drop and recreate the database, please always use the >>> >latest master - the clients, even if using an older version, should >>> >accomodate, we try keep the database schema changes backward compatible. >>> > >>> > >>> > >>> > >>> >For a short while, the "master" was broken due to a bad commit - you may >>> >have landed there with your master, can you please git pull lastest >>> >master ? >>> > >>> > >>> > >>> > >>> >About the specific bug you see - can you please let me know more about >>> >your setup ? >>> > >>> > >>> >- db engine you're using is mysql, right ? >>> > >>> > >>> >- commit ids for master and daisy that you're using ? >>> > >>> >- which commands do you use to start the server and the clients ? >>> > >>> > >>> > >>> >Cheers, >>> >Alex >>> > >>> > >>> > >>> > >>> >On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave < >>> [email protected]> >>> >wrote: >>> > >>> >Hi Craig, >>> > >>> >Did you delete your old database and recreate it using the current clone >>> >of poky as shown on >>> > >>> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster >>> >that is, running the database creation/migration commands that execute >>> in >>> > 'source toaster start'? >>> > >>> >-dave >>> > >>> >> -----Original Message----- >>> >> From: Neth, Craig Allen [mailto:[email protected]] >>> >> Sent: Thursday, July 03, 2014 4:05 PM >>> >> To: Lerner, Dave >>> >> Cc: [email protected] >>> >> Subject: Re: Need help with Production toaster setup >>> >> >>> >> Hi Dave, >>> >> >>> >> Thanks for the quick response! >>> >> >>> >> I had previous experienced the problem you mentioned and worked around >>> >>it >>> >> by adding a sleep between the —observe-only and the bitbake command. >>> >> Without the sleep, as you mention the events were just getting >>> dropped. >>> >> It’s nice to see a proper fix for that, however. >>> >> >>> >> But I don’t think that’s what’s happening now; from the tracebacks it >>> >> looks like the clients are sending events to the database but it’s not >>> >> setup the way they expect... >>> >> >>> >> My suspicion is that the schemas have changed but I don’t know how to >>> >>get >>> >> my database back into a state where it matches the schema the client >>> >> expects. >>> >> >>> >> Craig >>> >> >>> >> >>> >> >>> >> >>> >> On 7/3/14, 4:52 PM, "Lerner, Dave" <[email protected]> wrote: >>> >> >>> >> >Hi Craig, >>> >> > >>> >> >A patch submitted 15 minutes ago may solve your problem, it sounds >>> >> >similar to the problem that appeared after daisy in my script that >>> did: >>> >> > >>> >> >bitbake --server-only ... >>> >> >bitbake --observe-only ... ToasterUI >>> >> >bitbake <some-image> >>> >> > >>> >> >With recent releases the 'bitbake <some-image>' fires before the >>> >> >ToasterUI is ready (2nd command) to read events. The patch in >>> >> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a >>> message >>> >> >to the log file. You can change your script to block until that log >>> >> >message appears, before launching a build - if that's your problem. >>> >> > >>> >> >Hope this helps, >>> >> >Dave Lerner >>> >> > >>> >> >> -----Original Message----- >>> >> >> From: [email protected] >>> >> >>[mailto:[email protected]] On >>> >> >> Behalf Of Neth, Craig Allen >>> >> >> Sent: Thursday, July 03, 2014 3:34 PM >>> >> >> To: [email protected] >>> >> >> Subject: [Toaster] Need help with Production toaster setup >>> >> >> >>> >> >> A month or so ago, I set up a production toaster instance using the >>> >>poky >>> >> >> master branch. >>> >> >> Our projects were building with Daisy. Everything was working >>> fine. >>> >> >> >>> >> >> For various reasons, I needed to redo the production instance and >>> >>that >>> >> >> involved recloning the poky master repo, and now I¹m having issuesŠ >>> >> >> >>> >> >> After recloning poky and restarting the server, all my builds with >>> >>the >>> >> >> build logging turned on started throwing weird errors complaining >>> >>like >>> >> >> this: >>> >> >> >>> >> >> ERROR: Field 'project_id' doesn't have a default value >>> >> >> <traceback omitted> >>> >> >> >>> >> >> >>> >> >> And : >>> >> >> >>> >> >> ERROR: >>> >> >> Traceback (most recent call last): >>> >> >> File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main >>> >> >> buildinfohelper.store_started_task(event) >>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in >>> >> >> store_started_task >>> >> >> recipe_information = >>> >> >> self._get_recipe_information_from_taskfile(event.taskfile) >>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in >>> >> >> _get_recipe_information_from_taskfile >>> >> >> layer_version_obj = >>> >>self._get_layer_version_for_path(localfilepath) >>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in >>> >> >> _get_layer_version_for_path >>> >> >> assert 'build' in self.internal_state >>> >> >> AssertionError >>> >> >> >>> >> >> So I figured that there was probably some difference of opinion >>> >>between >>> >> >> the client and the server about the schema; I tried running the >>> >>Œsyncdb¹ >>> >> >> and Œmigrate orm¹ commands again on the server but that didn¹t >>> help. >>> >> >> >>> >> >> >>> >> >> --- >>> >> >> >>> >> >> So next I figured I would just delete the Œtoaster¹ database and >>> >>start >>> >> >> over. I tried going into mysql and dropping the toaster database, >>> >> >> recreating it, and then redoing the two python commands, but that >>> >> >>results >>> >> >> in this error from the migrate orm command: >>> >> >> >>> >> >> python bitbake/lib/toaster/manage.py migrate orm >>> >> >> Running migrations for orm: >>> >> >> - Migrating forwards to 0007_auto__add_helptext. >>> >> >> > orm:0001_initial >>> >> >> > orm:0002_auto__add_field_build_timespent >>> >> >> > orm:0003_timespent >>> >> >> - Migration 'orm:0003_timespent' is marked for no-dry-run. >>> >> >> > orm:0004_auto__add_field_package_installed_name >>> >> >> > >>> >> >> >>> >> >>> >>> >>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab >>> >>>>le >>> >> >>hi >>> >> >> stor >>> >> >> FATAL ERROR - The following SQL query failed: ALTER TABLE >>> `orm_build` >>> >> >>ADD >>> >> >> COLUMN `timespent` integer NOT NULL DEFAULT 0; >>> >> >> The error was: (1060, "Duplicate column name 'timespent'") >>> >> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and >>> >>its >>> >> >> values cannot be restored. >>> >> >> >>> >> >> This was all done while working on the daisy branch. >>> >> >> >>> >> >> So, then I tried the delete/recreate database trick again and >>> changed >>> >> >>back >>> >> >> to the master branch. Now I can run the two python commands ok, >>> but >>> >>my >>> >> >> daisy clients are back to throwing the assert errors... >>> >> >> >>> >> >> So, a few questions: >>> >> >> >>> >> >> >>> >> >> - Does it still work to run the production server on the master and >>> >> >> clients running daisy, or do I have to use the daisy version of the >>> >> >> production server, or do I need something newer than daisy but not >>> as >>> >> >>new >>> >> >> as master? >>> >> >> >>> >> >> - If for whatever reason I need to recreate the database, what are >>> >>the >>> >> >> exact steps that I need to do to delete/recreate? >>> >> >> It seems like just dropping the Œtoaster¹ database and recreating >>> it >>> >>is >>> >> >> insufficient, but I am not sure what else to do. >>> >> >> >>> >> >> The wiki pages are a bit vague on all this stuff, thanks for any >>> >>help. >>> >> >> >>> >> >> Craig >>> >> >> >>> >> >> -- >>> >> >> _______________________________________________ >>> >> >> toaster mailing list >>> >> >> [email protected] >>> >> >> https://lists.yoctoproject.org/listinfo/toaster >>> > >>> >-- >>> >_______________________________________________ >>> >toaster mailing list >>> >[email protected] >>> >https://lists.yoctoproject.org/listinfo/toaster >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >-- >>> >Alex DamianYocto Project >>> > >>> >SSG / OTC >>> >>> >> >> >> -- >> Alex Damian >> Yocto Project >> SSG / OTC >> >> > > > -- > Alex Damian > Yocto Project > SSG / OTC > -- Alex Damian Yocto Project SSG / OTC
-- _______________________________________________ toaster mailing list [email protected] https://lists.yoctoproject.org/listinfo/toaster
