Any better with Milton's 'self.start_time = datetime.utcnow()' change? Thanks, Chris
On Fri, 15 May 2020, Brian Dickson wrote: > On Fri, May 15, 2020 at 12:57 PM Brian Dickson > <brian.peter.dick...@gmail.com> wrote: > The local database is empty. I'll make sure that's the case just to be > safe. > > # peeringdb --version > > peeringdb 1.0.0 > > > # django-admin --version > > 2.2.12 > > > # pip freeze > > asgiref==3.2.7 > > certifi==2020.4.5.1 > > cfu==1.5.0 > > chardet==3.0.4 > > click==7.1.2 > > Django==2.2.12 > > django-countries==6.1.2 > > django-handleref==0.5.0 > > django-inet==0.4.0 > > django-peeringdb==2.0.0 > > future==0.18.2 > > idna==2.9 > > munge==1.0.0 > > mysqlclient==1.4.4 > > peeringdb==1.0.0 > > PyMySQL==0.9.3 > > pytz==2020.1 > > PyYAML==5.3.1 > > requests==2.23.0 > > six==1.14.0 > > sqlparse==0.3.1 > > twentyc.rpc==0.4.0 > > urllib3==1.25.9 > > > > Here is the result when running the sync on a clean empty local database: > > bash-3.2# peeringdb drop-tables > > bash-3.2# peeringdb sync > > Syncing to https://www.peeringdb.com/api > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc > > Fetching & updating all: org > > Updates to be processed: 18439 > > Ignoring object updated after sync began: (org-6864) > > Ignoring object updated after sync began: (org-13687) > > Ignoring object updated after sync began: (org-23582) > > Ignoring object updated after sync began: (org-26092) > > Ignoring object updated after sync began: (org-26094) > > Ignoring object updated after sync began: (org-26095) > > Ignoring object updated after sync began: (org-26096) > > Ignoring object updated after sync began: (org-26097) > > Ignoring object updated after sync began: (org-26098) > > Fetching & updating all: fac > > Updates to be processed: 3714 > > Ignoring object updated after sync began: (fac-3634) > > Ignoring object updated after sync began: (fac-8262) > > Ignoring object updated after sync began: (fac-8582) > > Ignoring object updated after sync began: (fac-8583) > > Ignoring object updated after sync began: (fac-8584) > > Ignoring object updated after sync began: (fac-8585) > > Ignoring object updated after sync began: (fac-8586) > > Ignoring object updated after sync began: (fac-8587) > > Ignoring object updated after sync began: (fac-8588) > > Ignoring object updated after sync began: (fac-8591) > > Ignoring object updated after sync began: (fac-8592) > > Ignoring object updated after sync began: (fac-8593) > > Ignoring object updated after sync began: (fac-8594) > > Ignoring object updated after sync began: (fac-8595) > > Ignoring object updated after sync began: (fac-8597) > > Ignoring object updated after sync began: (fac-8598) > > Ignoring object updated after sync began: (fac-8599) > > Ignoring object updated after sync began: (fac-8600) > > Ignoring object updated after sync began: (fac-8601) > > Ignoring object updated after sync began: (fac-8602) > > Ignoring object updated after sync began: (fac-8603) > > Fetching & updating all: net > > Updates to be processed: 18845 > > Ignoring object updated after sync began: (net-2913) > > Ignoring object updated after sync began: (net-3087) > > Ignoring object updated after sync began: (net-4823) > > Ignoring object updated after sync began: (net-5442) > > Ignoring object updated after sync began: (net-7482) > > Ignoring object updated after sync began: (net-9435) > > Ignoring object updated after sync began: (net-13681) > > Ignoring object updated after sync began: (net-14816) > > Ignoring object updated after sync began: (net-19500) > > Ignoring object updated after sync began: (net-20575) > > Ignoring object updated after sync began: (net-21292) > > Ignoring object updated after sync began: (net-21916) > > Ignoring object updated after sync began: (net-23222) > > Ignoring object updated after sync began: (net-23266) > > Ignoring object updated after sync began: (net-23280) > > Ignoring object updated after sync began: (net-23283) > > Ignoring object updated after sync began: (net-23284) > > Ignoring object updated after sync began: (net-23285) > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in <module> > > sys.exit(main()) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", > line 62, in main > > return handler(config=cfg, **vars(options)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 20, in _wrapped > > r = func(*a, **k) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 219, in handle > > client.update_all(rs) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, in update_all > > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 78, in _atomic_update > > sync_func() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, in <lambda> > > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line 65, in _wrapped > > return loop.run_until_complete(func(*a, **k)) > > File > "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", > line 488, in run_until_complete > > return future.result() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line 41, in _wrapped > > item = gen.send(r) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 287, in sync_row > > B.clean(obj) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py", > line 145, in clean > > obj.full_clean() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", > line 1203, in full_clean > > raise ValidationError(errors) > > django.core.exceptions.ValidationError: {'org': ['organization instance with > id 13687 does not exist.']} > > > > On Fri, May 15, 2020 at 10:55 AM Chris Caputo <ccap...@alt.net> wrote: > Hi. Does restarting with a fresh/empty local database help at all? > > Also, what does the following show?: > > peeringdb --version > django-admin --version > pip freeze > > Thanks, > Chris > > On Fri, 15 May 2020, Brian Dickson wrote: > > Arnold Nipper said to send to this list. > > > > (This is kind of urgent for me, my goal is to get a snapshot of the > data, so any alternative workaround would be helpful in the > > meantime.) > > > > I'm in the process of setting up a local copy, and have run into a > problem after getting the peeringdb-py stuff set up. > > > > When I do peeringdb sync (with the correct mysql database UTF8 stuff > and everything,), I am currently getting a persistent error > > which aborts the sync with no data in the tables: > > > > (Apologies in advance for the long error output text.) > > Syncing to https://www.peeringdb.com/api > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan > poc > > Fetching & updating all: org > > Updates to be processed: 5 > > Ignoring object updated after sync began: (org-26015) > > Ignoring object updated after sync began: (org-26052) > > Ignoring object updated after sync began: (org-17918) > > Ignoring object updated after sync began: (org-26053) > > Fetching & updating all: fac > > Updates to be processed: 0 > > Fetching & updating all: net > > Updates to be processed: 18800 > > Ignoring object updated after sync began: (net-1356) > > Ignoring object updated after sync began: (net-3684) > > Ignoring object updated after sync began: (net-7924) > > Ignoring object updated after sync began: (net-10733) > > Ignoring object updated after sync began: (net-13084) > > Ignoring object updated after sync began: (net-14581) > > Ignoring object updated after sync began: (net-15702) > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 84, in _execute > > return self.cursor.execute(sql, params) > > File > > > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", > line > > 71, in execute > > return self.cursor.execute(query, args) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 209, in > > execute > > res = self._query(query) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 315, in > > _query > > db.query(q) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", > line > > 226, in query > > _mysql.connection.query(self, query) > > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a > child row: a foreign key constraint fails > > (`peeringdb`.`peeringdb_network`, CONSTRAINT > `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') > > > > The above exception was the direct cause of the following exception: > > > > Traceback (most recent call last): > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", > line 8, in <module> > > sys.exit(main()) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", > line 62, in > > main > > return handler(config=cfg, **vars(options)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 20, > > in _wrapped > > r = func(*a, **k) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", > line 219, > > in handle > > client.update_all(rs) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, > > in update_all > > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 78, > > in _atomic_update > > sync_func() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 66, > > in <lambda> > > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line > > 65, in _wrapped > > return loop.run_until_complete(func(*a, **k)) > > File > "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", > line 488, in > > run_until_complete > > return future.result() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", > line > > 41, in _wrapped > > item = gen.send(r) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", > line 274, > > in sync_row > > B.save(obj) > > > File"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend > > .py", line 148, in save > > obj.save() > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", > line > > 741, in save > > force_update=force_update, update_fields=update_fields) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", > line > > 779, in save_base > > force_update, using, update_fields, > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", > line > > 870, in _save_table > > result = self._do_insert(cls._base_manager, using, fields, > update_pk, raw) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", > line > > 908, in _do_insert > > using=using, raw=raw) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/manager.py", > > line 82, in manager_method > > return getattr(self.get_queryset(), name)(*args, **kwargs) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/query.py", > line > > 1186, in _insert > > return query.get_compiler(using=using).execute_sql(return_id) > > File > > > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", > line > > 1375, in execute_sql > > cursor.execute(sql, params) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 67, in execute > > return self._execute_with_wrappers(sql, params, many=False, > executor=self._execute) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 76, in _execute_with_wrappers > > return executor(sql, params, many, context) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 84, in _execute > > return self.cursor.execute(sql, params) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/utils.py", > line 89, in > > __exit__ > > raise dj_exc_value.with_traceback(traceback) from exc_value > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", > > line 84, in _execute > > return self.cursor.execute(sql, params) > > File > > > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", > line > > 71, in execute > > return self.cursor.execute(query, args) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 209, in > > execute > > res = self._query(query) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", > line 315, in > > _query > > db.query(q) > > File > "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", > line > > 226, in query > > _mysql.connection.query(self, query) > > django.db.utils.IntegrityError: (1452, 'Cannot add or update a child > row: a foreign key constraint fails > > (`peeringdb`.`peeringdb_network`, CONSTRAINT > `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') > > > > Please let me know if this is a problem on my end, or a problem with > the data in the master db itself... > > > > If there is something I should do to fix this on my end, any advice > would be greatly appreciated. > > (If there is any other way to grab a copy of the tables in mysql > format, that's all I'm really trying to get, just a snapshot of > > the current peeringdb database to process with some perl scripts I > have.) > > > > Thanks in advance, > > > > Brian Dickson > > > > > > > > >
_______________________________________________ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech