Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
If you are situated West of UTC you might be running into the timezone issue I reported. The work around I had was to change this line<https://github.com/peeringdb/peeringdb-py/blob/289524928c62115da114a0a05e591c940b03ac4f/peeringdb/_update.py#L97> in peeringdb/_update.py from self.start_time = datetime.now() to self.start_time = datetime.utcnow() From: Pdb-tech on behalf of Brian Dickson Sent: Friday, May 15, 2020 10:43:44 AM To: pdb-tech@lists.peeringdb.com Subject: Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails? Arnold Nipper said to send to this list. (This is kind of urgent for me, my goal is to get a snapshot of the data, so any alternative workaround would be helpful in the meantime.) I'm in the process of setting up a local copy, and have run into a problem after getting the peeringdb-py stuff set up. When I do peeringdb sync (with the correct mysql database UTF8 stuff and everything,), I am currently getting a persistent error which aborts the sync with no data in the tables: (Apologies in advance for the long error output text.) Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 5 Ignoring object updated after sync began: (org-26015) Ignoring object updated after sync began: (org-26052) Ignoring object updated after sync began: (org-17918) Ignoring object updated after sync began: (org-26053) Fetching & updating all: fac Updates to be processed: 0 Fetching & updating all: net Updates to be processed: 18800 Ignoring object updated after sync began: (net-1356) Ignoring object updated after sync began: (net-3684) Ignoring object updated after sync began: (net-7924) Ignoring object updated after sync began: (net-10733) Ignoring object updated after sync began: (net-13084) Ignoring object updated after sync began: (net-14581) Ignoring object updated after sync began: (net-15702) Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute return self.cursor.execute(query, args) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute res = self._query(query) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query db.query(q) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query _mysql.connection.query(self, query) MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`peeringdb`.`peeringdb_network`, CONSTRAINT `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY (`org_id`) REFERENCES `peeringdb_organization` (`id`))') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", line 8, in sys.exit(main()) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped return loop.run_until_complete(func(*a, **k)) File "/opt/local/Library/Frameworks/Python.fra
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
And yes, it is now sync'd. Thanks again! Brian On Fri, May 15, 2020 at 2:55 PM Brian Dickson wrote: > Yes, it's still running, but I'm not getting the error messages I was > seeing before. > I expect it to work completely now. > > Thanks, everyone, especially Milton! > > (Also, you may want to approve his patch. :-) ) > > Brian > > On Fri, May 15, 2020 at 2:42 PM Chris Caputo wrote: > >> Any better with Milton's 'self.start_time = datetime.utcnow()' change? >> >> Thanks, >> Chris >> >> On Fri, 15 May 2020, Brian Dickson wrote: >> > On Fri, May 15, 2020 at 12:57 PM Brian Dickson < >> brian.peter.dick...@gmail.com> wrote: >> > The local database is empty. I'll make sure that's the case just >> to be safe. >> > >> > # peeringdb --version >> > >> > peeringdb 1.0.0 >> > >> > >> > # django-admin --version >> > >> > 2.2.12 >> > >> > >> > # pip freeze >> > >> > asgiref==3.2.7 >> > >> > certifi==2020.4.5.1 >> > >> > cfu==1.5.0 >> > >> > chardet==3.0.4 >> > >> > click==7.1.2 >> > >> > Django==2.2.12 >> > >> > django-countries==6.1.2 >> > >> > django-handleref==0.5.0 >> > >> > django-inet==0.4.0 >> > >> > django-peeringdb==2.0.0 >> > >> > future==0.18.2 >> > >> > idna==2.9 >> > >> > munge==1.0.0 >> > >> > mysqlclient==1.4.4 >> > >> > peeringdb==1.0.0 >> > >> > PyMySQL==0.9.3 >> > >> > pytz==2020.1 >> > >> > PyYAML==5.3.1 >> > >> > requests==2.23.0 >> > >> > six==1.14.0 >> > >> > sqlparse==0.3.1 >> > >> > twentyc.rpc==0.4.0 >> > >> > urllib3==1.25.9 >> > >> > >> > >> > Here is the result when running the sync on a clean empty local >> database: >> > >> > bash-3.2# peeringdb drop-tables >> > >> > bash-3.2# peeringdb sync >> > >> > Syncing to https://www.peeringdb.com/api >> > >> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc >> > >> > Fetching & updating all: org >> > >> > Updates to be processed: 18439 >> > >> > Ignoring object updated after sync began: (org-6864) >> > >> > Ignoring object updated after sync began: (org-13687) >> > >> > Ignoring object updated after sync began: (org-23582) >> > >> > Ignoring object updated after sync began: (org-26092) >> > >> > Ignoring object updated after sync began: (org-26094) >> > >> > Ignoring object updated after sync began: (org-26095) >> > >> > Ignoring object updated after sync began: (org-26096) >> > >> > Ignoring object updated after sync began: (org-26097) >> > >> > Ignoring object updated after sync began: (org-26098) >> > >> > Fetching & updating all: fac >> > >> > Updates to be processed: 3714 >> > >> > Ignoring object updated after sync began: (fac-3634) >> > >> > Ignoring object updated after sync began: (fac-8262) >> > >> > Ignoring object updated after sync began: (fac-8582) >> > >> > Ignoring object updated after sync began: (fac-8583) >> > >> > Ignoring object updated after sync began: (fac-8584) >> > >> > Ignoring object updated after sync began: (fac-8585) >> > >> > Ignoring object updated after sync began: (fac-8586) >> > >> > Ignoring object updated after sync began: (fac-8587) >> > >> > Ignoring object updated after sync began: (fac-8588) >> > >> > Ignoring object updated after sync began: (fac-8591) >> > >> > Ignoring object updated after sync began: (fac-8592) >> > >> > Ignoring object updated after sync began: (fac-8593) >> > >> > Ignoring object updated after sync began: (fac-8594) >> > >> > Ignoring object updated after sync began: (fac-8595) >> > >> > Ignoring object updated after sync began: (fac-8597) >> > >> > Ignoring object updated after sync began: (fac-8598) >> > >> > Ignoring object updated after sync began: (fac-8599) >> > >> > Ignoring object updated after sync began: (fac-8600) >> > >> > Ignoring object updated after sync began: (fac-8601) >> > >> > Ignoring object updated after sync began: (fac-8602) >> > >> > Ignoring object updated after sync began: (fac-8603) >> > >> > Fetching & updating all: net >> > >> > Updates to be processed: 18845 >> > >> > Ignoring object updated after sync began: (net-2913) >> > >> > Ignoring object updated after sync began: (net-3087) >> > >> > Ignoring object updated after sync began: (net-4823) >> > >> > Ignoring object updated after sync began: (net-5442) >> > >> > Ignoring object updated after sync began: (net-7482) >> > >> > Ignoring object updated after sync began: (net-9435) >> > >> > Ignoring object updated after sync began: (net-13681) >> > >> > Ignoring object updated after sync began: (net-14816) >> > >> > Ignoring object updated after sync began: (net-19500) >> > >> > Ignoring object updated after sync began: (net-20575) >> > >> > Ignoring object updated after sync began: (net-21292) >> > >> > Ignoring object updated after sync began: (net-21916) >> > >> > Ignoring object updated after sync began: (net-23222) >> > >> > Ignoring object updated after sync began: (net-23266) >> > >> > Ignoring object updated after sync began: (net-23280) >> > >> > Ignoring object updated after sync began: (net-23283) >> > >> > Ignoring object updated after sync began: (net-23284
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Yes, it's still running, but I'm not getting the error messages I was seeing before. I expect it to work completely now. Thanks, everyone, especially Milton! (Also, you may want to approve his patch. :-) ) Brian On Fri, May 15, 2020 at 2:42 PM Chris Caputo wrote: > Any better with Milton's 'self.start_time = datetime.utcnow()' change? > > Thanks, > Chris > > On Fri, 15 May 2020, Brian Dickson wrote: > > On Fri, May 15, 2020 at 12:57 PM Brian Dickson < > brian.peter.dick...@gmail.com> wrote: > > The local database is empty. I'll make sure that's the case just > to be safe. > > > > # peeringdb --version > > > > peeringdb 1.0.0 > > > > > > # django-admin --version > > > > 2.2.12 > > > > > > # pip freeze > > > > asgiref==3.2.7 > > > > certifi==2020.4.5.1 > > > > cfu==1.5.0 > > > > chardet==3.0.4 > > > > click==7.1.2 > > > > Django==2.2.12 > > > > django-countries==6.1.2 > > > > django-handleref==0.5.0 > > > > django-inet==0.4.0 > > > > django-peeringdb==2.0.0 > > > > future==0.18.2 > > > > idna==2.9 > > > > munge==1.0.0 > > > > mysqlclient==1.4.4 > > > > peeringdb==1.0.0 > > > > PyMySQL==0.9.3 > > > > pytz==2020.1 > > > > PyYAML==5.3.1 > > > > requests==2.23.0 > > > > six==1.14.0 > > > > sqlparse==0.3.1 > > > > twentyc.rpc==0.4.0 > > > > urllib3==1.25.9 > > > > > > > > Here is the result when running the sync on a clean empty local database: > > > > bash-3.2# peeringdb drop-tables > > > > bash-3.2# peeringdb sync > > > > Syncing to https://www.peeringdb.com/api > > > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc > > > > Fetching & updating all: org > > > > Updates to be processed: 18439 > > > > Ignoring object updated after sync began: (org-6864) > > > > Ignoring object updated after sync began: (org-13687) > > > > Ignoring object updated after sync began: (org-23582) > > > > Ignoring object updated after sync began: (org-26092) > > > > Ignoring object updated after sync began: (org-26094) > > > > Ignoring object updated after sync began: (org-26095) > > > > Ignoring object updated after sync began: (org-26096) > > > > Ignoring object updated after sync began: (org-26097) > > > > Ignoring object updated after sync began: (org-26098) > > > > Fetching & updating all: fac > > > > Updates to be processed: 3714 > > > > Ignoring object updated after sync began: (fac-3634) > > > > Ignoring object updated after sync began: (fac-8262) > > > > Ignoring object updated after sync began: (fac-8582) > > > > Ignoring object updated after sync began: (fac-8583) > > > > Ignoring object updated after sync began: (fac-8584) > > > > Ignoring object updated after sync began: (fac-8585) > > > > Ignoring object updated after sync began: (fac-8586) > > > > Ignoring object updated after sync began: (fac-8587) > > > > Ignoring object updated after sync began: (fac-8588) > > > > Ignoring object updated after sync began: (fac-8591) > > > > Ignoring object updated after sync began: (fac-8592) > > > > Ignoring object updated after sync began: (fac-8593) > > > > Ignoring object updated after sync began: (fac-8594) > > > > Ignoring object updated after sync began: (fac-8595) > > > > Ignoring object updated after sync began: (fac-8597) > > > > Ignoring object updated after sync began: (fac-8598) > > > > Ignoring object updated after sync began: (fac-8599) > > > > Ignoring object updated after sync began: (fac-8600) > > > > Ignoring object updated after sync began: (fac-8601) > > > > Ignoring object updated after sync began: (fac-8602) > > > > Ignoring object updated after sync began: (fac-8603) > > > > Fetching & updating all: net > > > > Updates to be processed: 18845 > > > > Ignoring object updated after sync began: (net-2913) > > > > Ignoring object updated after sync began: (net-3087) > > > > Ignoring object updated after sync began: (net-4823) > > > > Ignoring object updated after sync began: (net-5442) > > > > Ignoring object updated after sync began: (net-7482) > > > > Ignoring object updated after sync began: (net-9435) > > > > Ignoring object updated after sync began: (net-13681) > > > > Ignoring object updated after sync began: (net-14816) > > > > Ignoring object updated after sync began: (net-19500) > > > > Ignoring object updated after sync began: (net-20575) > > > > Ignoring object updated after sync began: (net-21292) > > > > Ignoring object updated after sync began: (net-21916) > > > > Ignoring object updated after sync began: (net-23222) > > > > Ignoring object updated after sync began: (net-23266) > > > > Ignoring object updated after sync began: (net-23280) > > > > Ignoring object updated after sync began: (net-23283) > > > > Ignoring object updated after sync began: (net-23284) > > > > Ignoring object updated after sync began: (net-23285) > > > > Traceback (most recent call last): > > > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in > > > > sys.exit(main()) > > > > File > "/Users/bdickson1/Documents/projects/site-selection/p
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Any better with Milton's 'self.start_time = datetime.utcnow()' change? Thanks, Chris On Fri, 15 May 2020, Brian Dickson wrote: > On Fri, May 15, 2020 at 12:57 PM Brian Dickson > wrote: > The local database is empty. I'll make sure that's the case just to be > safe. > > # peeringdb --version > > peeringdb 1.0.0 > > > # django-admin --version > > 2.2.12 > > > # pip freeze > > asgiref==3.2.7 > > certifi==2020.4.5.1 > > cfu==1.5.0 > > chardet==3.0.4 > > click==7.1.2 > > Django==2.2.12 > > django-countries==6.1.2 > > django-handleref==0.5.0 > > django-inet==0.4.0 > > django-peeringdb==2.0.0 > > future==0.18.2 > > idna==2.9 > > munge==1.0.0 > > mysqlclient==1.4.4 > > peeringdb==1.0.0 > > PyMySQL==0.9.3 > > pytz==2020.1 > > PyYAML==5.3.1 > > requests==2.23.0 > > six==1.14.0 > > sqlparse==0.3.1 > > twentyc.rpc==0.4.0 > > urllib3==1.25.9 > > > > Here is the result when running the sync on a clean empty local database: > > bash-3.2# peeringdb drop-tables > > bash-3.2# peeringdb sync > > Syncing to https://www.peeringdb.com/api > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc > > Fetching & updating all: org > > Updates to be processed: 18439 > > Ignoring object updated after sync began: (org-6864) > > Ignoring object updated after sync began: (org-13687) > > Ignoring object updated after sync began: (org-23582) > > Ignoring object updated after sync began: (org-26092) > > Ignoring object updated after sync began: (org-26094) > > Ignoring object updated after sync began: (org-26095) > > Ignoring object updated after sync began: (org-26096) > > Ignoring object updated after sync began: (org-26097) > > Ignoring object updated after sync began: (org-26098) > > Fetching & updating all: fac > > Updates to be processed: 3714 > > Ignoring object updated after sync began: (fac-3634) > > Ignoring object updated after sync began: (fac-8262) > > Ignoring object updated after sync began: (fac-8582) > > Ignoring object updated after sync began: (fac-8583) > > Ignoring object updated after sync began: (fac-8584) > > Ignoring object updated after sync began: (fac-8585) > > Ignoring object updated after sync began: (fac-8586) > > Ignoring object updated after sync began: (fac-8587) > > Ignoring object updated after sync began: (fac-8588) > > Ignoring object updated after sync began: (fac-8591) > > Ignoring object updated after sync began: (fac-8592) > > Ignoring object updated after sync began: (fac-8593) > > Ignoring object updated after sync began: (fac-8594) > > Ignoring object updated after sync began: (fac-8595) > > Ignoring object updated after sync began: (fac-8597) > > Ignoring object updated after sync began: (fac-8598) > > Ignoring object updated after sync began: (fac-8599) > > Ignoring object updated after sync began: (fac-8600) > > Ignoring object updated after sync began: (fac-8601) > > Ignoring object updated after sync began: (fac-8602) > > Ignoring object updated after sync began: (fac-8603) > > Fetching & updating all: net > > Updates to be processed: 18845 > > Ignoring object updated after sync began: (net-2913) > > Ignoring object updated after sync began: (net-3087) > > Ignoring object updated after sync began: (net-4823) > > Ignoring object updated after sync began: (net-5442) > > Ignoring object updated after sync began: (net-7482) > > Ignoring object updated after sync began: (net-9435) > > Ignoring object updated after sync began: (net-13681) > > Ignoring object updated after sync began: (net-14816) > > Ignoring object updated after sync began: (net-19500) > > Ignoring object updated after sync began: (net-20575) > > Ignoring object updated after sync began: (net-21292) > > Ignoring object updated after sync began: (net-21916) > > Ignoring object updated after sync began: (net-23222) > > Ignoring object updated after sync began: (net-23266) > > Ignoring object updated after sync began: (net-23280) > > Ignoring object updated after sync began: (net-23283) > > Ignoring object updated after sync began: (net-23284) > > Ignoring object updated after sync began: (net-23285) > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in > > sys.exit(main()) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", > line 62, in main > > return handler(config=cfg, **vars(options)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 20, in _wrapped > > r = func(*a, **k) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 219, in handle > > client.update_all(rs) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Hi Brian, On 5/15/20 5:43 PM, Brian Dickson wrote: (This is kind of urgent for me, my goal is to get a snapshot of the data, so any alternative workaround would be helpful in the meantime.) Alternatively you could use: https://git.2e8.dk/peeringdb-simplesync/about/ -- Best regards Asbjørn Sloth Tønnesen ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
On Fri, May 15, 2020 at 12:57 PM Brian Dickson < brian.peter.dick...@gmail.com> wrote: > The local database is empty. I'll make sure that's the case just to be > safe. > > # peeringdb --version > > peeringdb 1.0.0 > > > # django-admin --version > > 2.2.12 > > > # pip freeze > > asgiref==3.2.7 > > certifi==2020.4.5.1 > > cfu==1.5.0 > > chardet==3.0.4 > > click==7.1.2 > > Django==2.2.12 > > django-countries==6.1.2 > > django-handleref==0.5.0 > > django-inet==0.4.0 > > django-peeringdb==2.0.0 > > future==0.18.2 > > idna==2.9 > > munge==1.0.0 > > mysqlclient==1.4.4 > > peeringdb==1.0.0 > > PyMySQL==0.9.3 > > pytz==2020.1 > > PyYAML==5.3.1 > > requests==2.23.0 > > six==1.14.0 > > sqlparse==0.3.1 > > twentyc.rpc==0.4.0 > > urllib3==1.25.9 > Here is the result when running the sync on a clean empty local database: bash-3.2# peeringdb drop-tables bash-3.2# peeringdb sync Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 18439 Ignoring object updated after sync began: (org-6864) Ignoring object updated after sync began: (org-13687) Ignoring object updated after sync began: (org-23582) Ignoring object updated after sync began: (org-26092) Ignoring object updated after sync began: (org-26094) Ignoring object updated after sync began: (org-26095) Ignoring object updated after sync began: (org-26096) Ignoring object updated after sync began: (org-26097) Ignoring object updated after sync began: (org-26098) Fetching & updating all: fac Updates to be processed: 3714 Ignoring object updated after sync began: (fac-3634) Ignoring object updated after sync began: (fac-8262) Ignoring object updated after sync began: (fac-8582) Ignoring object updated after sync began: (fac-8583) Ignoring object updated after sync began: (fac-8584) Ignoring object updated after sync began: (fac-8585) Ignoring object updated after sync began: (fac-8586) Ignoring object updated after sync began: (fac-8587) Ignoring object updated after sync began: (fac-8588) Ignoring object updated after sync began: (fac-8591) Ignoring object updated after sync began: (fac-8592) Ignoring object updated after sync began: (fac-8593) Ignoring object updated after sync began: (fac-8594) Ignoring object updated after sync began: (fac-8595) Ignoring object updated after sync began: (fac-8597) Ignoring object updated after sync began: (fac-8598) Ignoring object updated after sync began: (fac-8599) Ignoring object updated after sync began: (fac-8600) Ignoring object updated after sync began: (fac-8601) Ignoring object updated after sync began: (fac-8602) Ignoring object updated after sync began: (fac-8603) Fetching & updating all: net Updates to be processed: 18845 Ignoring object updated after sync began: (net-2913) Ignoring object updated after sync began: (net-3087) Ignoring object updated after sync began: (net-4823) Ignoring object updated after sync began: (net-5442) Ignoring object updated after sync began: (net-7482) Ignoring object updated after sync began: (net-9435) Ignoring object updated after sync began: (net-13681) Ignoring object updated after sync began: (net-14816) Ignoring object updated after sync began: (net-19500) Ignoring object updated after sync began: (net-20575) Ignoring object updated after sync began: (net-21292) Ignoring object updated after sync began: (net-21916) Ignoring object updated after sync began: (net-23222) Ignoring object updated after sync began: (net-23266) Ignoring object updated after sync began: (net-23280) Ignoring object updated after sync began: (net-23283) Ignoring object updated after sync began: (net-23284) Ignoring object updated after sync began: (net-23285) Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", line 8, in sys.exit(main()) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in self._atomic_update(lambda: c
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
The local database is empty. I'll make sure that's the case just to be safe. # peeringdb --version peeringdb 1.0.0 # django-admin --version 2.2.12 # pip freeze asgiref==3.2.7 certifi==2020.4.5.1 cfu==1.5.0 chardet==3.0.4 click==7.1.2 Django==2.2.12 django-countries==6.1.2 django-handleref==0.5.0 django-inet==0.4.0 django-peeringdb==2.0.0 future==0.18.2 idna==2.9 munge==1.0.0 mysqlclient==1.4.4 peeringdb==1.0.0 PyMySQL==0.9.3 pytz==2020.1 PyYAML==5.3.1 requests==2.23.0 six==1.14.0 sqlparse==0.3.1 twentyc.rpc==0.4.0 urllib3==1.25.9 On Fri, May 15, 2020 at 10:55 AM Chris Caputo wrote: > Hi. Does restarting with a fresh/empty local database help at all? > > Also, what does the following show?: > > peeringdb --version > django-admin --version > pip freeze > > Thanks, > Chris > > On Fri, 15 May 2020, Brian Dickson wrote: > > Arnold Nipper said to send to this list. > > > > (This is kind of urgent for me, my goal is to get a snapshot of the > data, so any alternative workaround would be helpful in the > > meantime.) > > > > I'm in the process of setting up a local copy, and have run into a > problem after getting the peeringdb-py stuff set up. > > > > When I do peeringdb sync (with the correct mysql database UTF8 stuff and > everything,), I am currently getting a persistent error > > which aborts the sync with no data in the tables: > > > > (Apologies in advance for the long error output text.) > > Syncing to https://www.peeringdb.com/api > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc > > Fetching & updating all: org > > Updates to be processed: 5 > > Ignoring object updated after sync began: (org-26015) > > Ignoring object updated after sync began: (org-26052) > > Ignoring object updated after sync began: (org-17918) > > Ignoring object updated after sync began: (org-26053) > > Fetching & updating all: fac > > Updates to be processed: 0 > > Fetching & updating all: net > > Updates to be processed: 18800 > > Ignoring object updated after sync began: (net-1356) > > Ignoring object updated after sync began: (net-3684) > > Ignoring object updated after sync began: (net-7924) > > Ignoring object updated after sync began: (net-10733) > > Ignoring object updated after sync began: (net-13084) > > Ignoring object updated after sync began: (net-14581) > > Ignoring object updated after sync began: (net-15702) > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 84, in _execute > > return self.cursor.execute(sql, params) > > File > > > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", > line > > 71, in execute > > return self.cursor.execute(query, args) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 209, in > > execute > > res = self._query(query) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 315, in > > _query > > db.query(q) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", > line > > 226, in query > > _mysql.connection.query(self, query) > > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child > row: a foreign key constraint fails > > (`peeringdb`.`peeringdb_network`, CONSTRAINT > `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') > > > > The above exception was the direct cause of the following exception: > > > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in > > sys.exit(main()) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", > line 62, in > > main > > return handler(config=cfg, **vars(options)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 20, > > in _wrapped > > r = func(*a, **k) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 219, > > in handle > > client.update_all(rs) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, > > in update_all > > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 78, > > in _atomic_update > > sync_func() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Hi. Does restarting with a fresh/empty local database help at all? Also, what does the following show?: peeringdb --version django-admin --version pip freeze Thanks, Chris On Fri, 15 May 2020, Brian Dickson wrote: > Arnold Nipper said to send to this list. > > (This is kind of urgent for me, my goal is to get a snapshot of the data, so > any alternative workaround would be helpful in the > meantime.) > > I'm in the process of setting up a local copy, and have run into a problem > after getting the peeringdb-py stuff set up. > > When I do peeringdb sync (with the correct mysql database UTF8 stuff and > everything,), I am currently getting a persistent error > which aborts the sync with no data in the tables: > > (Apologies in advance for the long error output text.) > Syncing to https://www.peeringdb.com/api > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc > Fetching & updating all: org > Updates to be processed: 5 > Ignoring object updated after sync began: (org-26015) > Ignoring object updated after sync began: (org-26052) > Ignoring object updated after sync began: (org-17918) > Ignoring object updated after sync began: (org-26053) > Fetching & updating all: fac > Updates to be processed: 0 > Fetching & updating all: net > Updates to be processed: 18800 > Ignoring object updated after sync began: (net-1356) > Ignoring object updated after sync began: (net-3684) > Ignoring object updated after sync began: (net-7924) > Ignoring object updated after sync began: (net-10733) > Ignoring object updated after sync began: (net-13084) > Ignoring object updated after sync began: (net-14581) > Ignoring object updated after sync began: (net-15702) > Traceback (most recent call last): > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > line 84, in _execute > return self.cursor.execute(sql, params) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", > line > 71, in execute > return self.cursor.execute(query, args) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 209, in > execute > res = self._query(query) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 315, in > _query > db.query(q) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", > line > 226, in query > _mysql.connection.query(self, query) > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: > a foreign key constraint fails > (`peeringdb`.`peeringdb_network`, CONSTRAINT > `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') > > The above exception was the direct cause of the following exception: > > Traceback (most recent call last): > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in > sys.exit(main()) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", > line 62, in > main > return handler(config=cfg, **vars(options)) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 20, > in _wrapped > r = func(*a, **k) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 219, > in handle > client.update_all(rs) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, > in update_all > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 78, > in _atomic_update > sync_func() > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, > in > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line > 65, in _wrapped > return loop.run_until_complete(func(*a, **k)) > File > "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", > line 488, in > run_until_complete > return future.result() > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line > 41, in _wrapped > item = gen.send(r) > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/l
Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?
Arnold Nipper said to send to this list. (This is kind of urgent for me, my goal is to get a snapshot of the data, so any alternative workaround would be helpful in the meantime.) I'm in the process of setting up a local copy, and have run into a problem after getting the peeringdb-py stuff set up. When I do peeringdb sync (with the correct mysql database UTF8 stuff and everything,), I am currently getting a persistent error which aborts the sync with no data in the tables: (Apologies in advance for the long error output text.) Syncing to https://www.peeringdb.com/api Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc Fetching & updating all: org Updates to be processed: 5 Ignoring object updated after sync began: (org-26015) Ignoring object updated after sync began: (org-26052) Ignoring object updated after sync began: (org-17918) Ignoring object updated after sync began: (org-26053) Fetching & updating all: fac Updates to be processed: 0 Fetching & updating all: net Updates to be processed: 18800 Ignoring object updated after sync began: (net-1356) Ignoring object updated after sync began: (net-3684) Ignoring object updated after sync began: (net-7924) Ignoring object updated after sync began: (net-10733) Ignoring object updated after sync began: (net-13084) Ignoring object updated after sync began: (net-14581) Ignoring object updated after sync began: (net-15702) Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute return self.cursor.execute(query, args) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute res = self._query(query) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query db.query(q) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query _mysql.connection.query(self, query) MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`peeringdb`.`peeringdb_network`, CONSTRAINT `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY (`org_id`) REFERENCES `peeringdb_organization` (`id`))') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", line 8, in sys.exit(main()) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", line 62, in main return handler(config=cfg, **vars(options)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 20, in _wrapped r = func(*a, **k) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 219, in handle client.update_all(rs) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in update_all self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 78, in _atomic_update sync_func() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in self._atomic_update(lambda: ctx.sync_resource(r, since=since)) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 65, in _wrapped return loop.run_until_complete(func(*a, **k)) File "/opt/local/Library/Frameworks/Python.framework/ Versions/3.6/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete return future.result() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 41, in _wrapped item = gen.send(r) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 274, in sync_row B.save(obj) File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py", line 148, in save obj.save() File "/Users/bdickson1/Documents/projects/site-selection/ pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save force_update=force_update, update_fields=update_fields) File "/U