Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Milton Ngan 
If you are situated West of UTC you might be running into the timezone issue I 
reported. The work around I had was to change this 
line<https://github.com/peeringdb/peeringdb-py/blob/289524928c62115da114a0a05e591c940b03ac4f/peeringdb/_update.py#L97>
 in peeringdb/_update.py


from

self.start_time = datetime.now()


to


self.start_time = datetime.utcnow()




From: Pdb-tech  on behalf of Brian 
Dickson 
Sent: Friday, May 15, 2020 10:43:44 AM
To: pdb-tech@lists.peeringdb.com
Subject: Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key 
constraint fails?



Arnold Nipper said to send to this list.

(This is kind of urgent for me, my goal is to get a snapshot of the data, so 
any alternative workaround would be helpful in the meantime.)

I'm in the process of setting up a local copy, and have run into a problem 
after getting the peeringdb-py stuff set up.

When I do peeringdb sync (with the correct mysql database UTF8 stuff and 
everything,), I am currently getting a persistent error which aborts the sync 
with no data in the tables:

(Apologies in advance for the long error output text.)
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 5
Ignoring object updated after sync began: (org-26015)
Ignoring object updated after sync began: (org-26052)
Ignoring object updated after sync began: (org-17918)
Ignoring object updated after sync began: (org-26053)
Fetching & updating all: fac
Updates to be processed: 0
Fetching & updating all: net
Updates to be processed: 18800
Ignoring object updated after sync began: (net-1356)
Ignoring object updated after sync began: (net-3684)
Ignoring object updated after sync began: (net-7924)
Ignoring object updated after sync began: (net-10733)
Ignoring object updated after sync began: (net-13084)
Ignoring object updated after sync began: (net-14581)
Ignoring object updated after sync began: (net-15702)
Traceback (most recent call last):
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
 line 84, in _execute
return self.cursor.execute(sql, params)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
 line 71, in execute
return self.cursor.execute(query, args)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
 line 209, in execute
res = self._query(query)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
 line 315, in _query
db.query(q)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
 line 226, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: a 
foreign key constraint fails (`peeringdb`.`peeringdb_network`, CONSTRAINT 
`peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY 
(`org_id`) REFERENCES `peeringdb_organization` (`id`))')


The above exception was the direct cause of the following exception:


Traceback (most recent call last):
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", 
line 8, in 
sys.exit(main())
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
 line 62, in main
return handler(config=cfg, **vars(options))
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
 line 20, in _wrapped
r = func(*a, **k)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
 line 219, in handle
client.update_all(rs)
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
 line 66, in update_all
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
 line 78, in _atomic_update
sync_func()
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
 line 66, in 
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
 line 65, in _wrapped
return loop.run_until_complete(func(*a, **k))
  File 
"/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/p

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Brian Dickson 
And yes, it is now sync'd.

Thanks again!

Brian

On Fri, May 15, 2020 at 2:55 PM Brian Dickson 
wrote:

> Yes, it's still running, but I'm not getting the error messages I was
> seeing before.
> I expect it to work completely now.
>
> Thanks, everyone, especially Milton!
>
> (Also, you may want to approve his patch. :-) )
>
> Brian
>
> On Fri, May 15, 2020 at 2:42 PM Chris Caputo  wrote:
>
>> Any better with Milton's 'self.start_time = datetime.utcnow()' change?
>>
>> Thanks,
>> Chris
>>
>> On Fri, 15 May 2020, Brian Dickson wrote:
>> > On Fri, May 15, 2020 at 12:57 PM Brian Dickson <
>> brian.peter.dick...@gmail.com> wrote:
>> >   The local database is empty. I'll make sure that's the case just
>> to be safe.
>> >
>> > # peeringdb --version
>> >
>> > peeringdb 1.0.0
>> >
>> >
>> > # django-admin --version
>> >
>> > 2.2.12
>> >
>> >
>> > # pip freeze
>> >
>> > asgiref==3.2.7
>> >
>> > certifi==2020.4.5.1
>> >
>> > cfu==1.5.0
>> >
>> > chardet==3.0.4
>> >
>> > click==7.1.2
>> >
>> > Django==2.2.12
>> >
>> > django-countries==6.1.2
>> >
>> > django-handleref==0.5.0
>> >
>> > django-inet==0.4.0
>> >
>> > django-peeringdb==2.0.0
>> >
>> > future==0.18.2
>> >
>> > idna==2.9
>> >
>> > munge==1.0.0
>> >
>> > mysqlclient==1.4.4
>> >
>> > peeringdb==1.0.0
>> >
>> > PyMySQL==0.9.3
>> >
>> > pytz==2020.1
>> >
>> > PyYAML==5.3.1
>> >
>> > requests==2.23.0
>> >
>> > six==1.14.0
>> >
>> > sqlparse==0.3.1
>> >
>> > twentyc.rpc==0.4.0
>> >
>> > urllib3==1.25.9
>> >
>> >
>> >
>> > Here is the result when running the sync on a clean empty local
>> database:
>> >
>> > bash-3.2# peeringdb drop-tables
>> >
>> > bash-3.2# peeringdb sync
>> >
>> > Syncing to https://www.peeringdb.com/api
>> >
>> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
>> >
>> > Fetching & updating all: org
>> >
>> > Updates to be processed: 18439
>> >
>> > Ignoring object updated after sync began: (org-6864)
>> >
>> > Ignoring object updated after sync began: (org-13687)
>> >
>> > Ignoring object updated after sync began: (org-23582)
>> >
>> > Ignoring object updated after sync began: (org-26092)
>> >
>> > Ignoring object updated after sync began: (org-26094)
>> >
>> > Ignoring object updated after sync began: (org-26095)
>> >
>> > Ignoring object updated after sync began: (org-26096)
>> >
>> > Ignoring object updated after sync began: (org-26097)
>> >
>> > Ignoring object updated after sync began: (org-26098)
>> >
>> > Fetching & updating all: fac
>> >
>> > Updates to be processed: 3714
>> >
>> > Ignoring object updated after sync began: (fac-3634)
>> >
>> > Ignoring object updated after sync began: (fac-8262)
>> >
>> > Ignoring object updated after sync began: (fac-8582)
>> >
>> > Ignoring object updated after sync began: (fac-8583)
>> >
>> > Ignoring object updated after sync began: (fac-8584)
>> >
>> > Ignoring object updated after sync began: (fac-8585)
>> >
>> > Ignoring object updated after sync began: (fac-8586)
>> >
>> > Ignoring object updated after sync began: (fac-8587)
>> >
>> > Ignoring object updated after sync began: (fac-8588)
>> >
>> > Ignoring object updated after sync began: (fac-8591)
>> >
>> > Ignoring object updated after sync began: (fac-8592)
>> >
>> > Ignoring object updated after sync began: (fac-8593)
>> >
>> > Ignoring object updated after sync began: (fac-8594)
>> >
>> > Ignoring object updated after sync began: (fac-8595)
>> >
>> > Ignoring object updated after sync began: (fac-8597)
>> >
>> > Ignoring object updated after sync began: (fac-8598)
>> >
>> > Ignoring object updated after sync began: (fac-8599)
>> >
>> > Ignoring object updated after sync began: (fac-8600)
>> >
>> > Ignoring object updated after sync began: (fac-8601)
>> >
>> > Ignoring object updated after sync began: (fac-8602)
>> >
>> > Ignoring object updated after sync began: (fac-8603)
>> >
>> > Fetching & updating all: net
>> >
>> > Updates to be processed: 18845
>> >
>> > Ignoring object updated after sync began: (net-2913)
>> >
>> > Ignoring object updated after sync began: (net-3087)
>> >
>> > Ignoring object updated after sync began: (net-4823)
>> >
>> > Ignoring object updated after sync began: (net-5442)
>> >
>> > Ignoring object updated after sync began: (net-7482)
>> >
>> > Ignoring object updated after sync began: (net-9435)
>> >
>> > Ignoring object updated after sync began: (net-13681)
>> >
>> > Ignoring object updated after sync began: (net-14816)
>> >
>> > Ignoring object updated after sync began: (net-19500)
>> >
>> > Ignoring object updated after sync began: (net-20575)
>> >
>> > Ignoring object updated after sync began: (net-21292)
>> >
>> > Ignoring object updated after sync began: (net-21916)
>> >
>> > Ignoring object updated after sync began: (net-23222)
>> >
>> > Ignoring object updated after sync began: (net-23266)
>> >
>> > Ignoring object updated after sync began: (net-23280)
>> >
>> > Ignoring object updated after sync began: (net-23283)
>> >
>> > Ignoring object updated after sync began:

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Brian Dickson 
Yes, it's still running, but I'm not getting the error messages I was
seeing before.
I expect it to work completely now.

Thanks, everyone, especially Milton!

(Also, you may want to approve his patch. :-) )

Brian

On Fri, May 15, 2020 at 2:42 PM Chris Caputo  wrote:

> Any better with Milton's 'self.start_time = datetime.utcnow()' change?
>
> Thanks,
> Chris
>
> On Fri, 15 May 2020, Brian Dickson wrote:
> > On Fri, May 15, 2020 at 12:57 PM Brian Dickson <
> brian.peter.dick...@gmail.com> wrote:
> >   The local database is empty. I'll make sure that's the case just
> to be safe.
> >
> > # peeringdb --version
> >
> > peeringdb 1.0.0
> >
> >
> > # django-admin --version
> >
> > 2.2.12
> >
> >
> > # pip freeze
> >
> > asgiref==3.2.7
> >
> > certifi==2020.4.5.1
> >
> > cfu==1.5.0
> >
> > chardet==3.0.4
> >
> > click==7.1.2
> >
> > Django==2.2.12
> >
> > django-countries==6.1.2
> >
> > django-handleref==0.5.0
> >
> > django-inet==0.4.0
> >
> > django-peeringdb==2.0.0
> >
> > future==0.18.2
> >
> > idna==2.9
> >
> > munge==1.0.0
> >
> > mysqlclient==1.4.4
> >
> > peeringdb==1.0.0
> >
> > PyMySQL==0.9.3
> >
> > pytz==2020.1
> >
> > PyYAML==5.3.1
> >
> > requests==2.23.0
> >
> > six==1.14.0
> >
> > sqlparse==0.3.1
> >
> > twentyc.rpc==0.4.0
> >
> > urllib3==1.25.9
> >
> >
> >
> > Here is the result when running the sync on a clean empty local database:
> >
> > bash-3.2# peeringdb drop-tables
> >
> > bash-3.2# peeringdb sync
> >
> > Syncing to https://www.peeringdb.com/api
> >
> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> >
> > Fetching & updating all: org
> >
> > Updates to be processed: 18439
> >
> > Ignoring object updated after sync began: (org-6864)
> >
> > Ignoring object updated after sync began: (org-13687)
> >
> > Ignoring object updated after sync began: (org-23582)
> >
> > Ignoring object updated after sync began: (org-26092)
> >
> > Ignoring object updated after sync began: (org-26094)
> >
> > Ignoring object updated after sync began: (org-26095)
> >
> > Ignoring object updated after sync began: (org-26096)
> >
> > Ignoring object updated after sync began: (org-26097)
> >
> > Ignoring object updated after sync began: (org-26098)
> >
> > Fetching & updating all: fac
> >
> > Updates to be processed: 3714
> >
> > Ignoring object updated after sync began: (fac-3634)
> >
> > Ignoring object updated after sync began: (fac-8262)
> >
> > Ignoring object updated after sync began: (fac-8582)
> >
> > Ignoring object updated after sync began: (fac-8583)
> >
> > Ignoring object updated after sync began: (fac-8584)
> >
> > Ignoring object updated after sync began: (fac-8585)
> >
> > Ignoring object updated after sync began: (fac-8586)
> >
> > Ignoring object updated after sync began: (fac-8587)
> >
> > Ignoring object updated after sync began: (fac-8588)
> >
> > Ignoring object updated after sync began: (fac-8591)
> >
> > Ignoring object updated after sync began: (fac-8592)
> >
> > Ignoring object updated after sync began: (fac-8593)
> >
> > Ignoring object updated after sync began: (fac-8594)
> >
> > Ignoring object updated after sync began: (fac-8595)
> >
> > Ignoring object updated after sync began: (fac-8597)
> >
> > Ignoring object updated after sync began: (fac-8598)
> >
> > Ignoring object updated after sync began: (fac-8599)
> >
> > Ignoring object updated after sync began: (fac-8600)
> >
> > Ignoring object updated after sync began: (fac-8601)
> >
> > Ignoring object updated after sync began: (fac-8602)
> >
> > Ignoring object updated after sync began: (fac-8603)
> >
> > Fetching & updating all: net
> >
> > Updates to be processed: 18845
> >
> > Ignoring object updated after sync began: (net-2913)
> >
> > Ignoring object updated after sync began: (net-3087)
> >
> > Ignoring object updated after sync began: (net-4823)
> >
> > Ignoring object updated after sync began: (net-5442)
> >
> > Ignoring object updated after sync began: (net-7482)
> >
> > Ignoring object updated after sync began: (net-9435)
> >
> > Ignoring object updated after sync began: (net-13681)
> >
> > Ignoring object updated after sync began: (net-14816)
> >
> > Ignoring object updated after sync began: (net-19500)
> >
> > Ignoring object updated after sync began: (net-20575)
> >
> > Ignoring object updated after sync began: (net-21292)
> >
> > Ignoring object updated after sync began: (net-21916)
> >
> > Ignoring object updated after sync began: (net-23222)
> >
> > Ignoring object updated after sync began: (net-23266)
> >
> > Ignoring object updated after sync began: (net-23280)
> >
> > Ignoring object updated after sync began: (net-23283)
> >
> > Ignoring object updated after sync began: (net-23284)
> >
> > Ignoring object updated after sync began: (net-23285)
> >
> > Traceback (most recent call last):
> >
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
> line 8, in 
> >
> > sys.exit(main())
> >
> >   File
>

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Chris Caputo 
Any better with Milton's 'self.start_time = datetime.utcnow()' change?

Thanks,
Chris

On Fri, 15 May 2020, Brian Dickson wrote:
> On Fri, May 15, 2020 at 12:57 PM Brian Dickson 
>  wrote:
>   The local database is empty. I'll make sure that's the case just to be 
> safe.
> 
> # peeringdb --version
> 
> peeringdb 1.0.0
> 
> 
> # django-admin --version
> 
> 2.2.12
> 
> 
> # pip freeze
> 
> asgiref==3.2.7
> 
> certifi==2020.4.5.1
> 
> cfu==1.5.0
> 
> chardet==3.0.4
> 
> click==7.1.2
> 
> Django==2.2.12
> 
> django-countries==6.1.2
> 
> django-handleref==0.5.0
> 
> django-inet==0.4.0
> 
> django-peeringdb==2.0.0
> 
> future==0.18.2
> 
> idna==2.9
> 
> munge==1.0.0
> 
> mysqlclient==1.4.4
> 
> peeringdb==1.0.0
> 
> PyMySQL==0.9.3
> 
> pytz==2020.1
> 
> PyYAML==5.3.1
> 
> requests==2.23.0
> 
> six==1.14.0
> 
> sqlparse==0.3.1
> 
> twentyc.rpc==0.4.0
> 
> urllib3==1.25.9
> 
> 
> 
> Here is the result when running the sync on a clean empty local database:
> 
> bash-3.2# peeringdb drop-tables
> 
> bash-3.2# peeringdb sync
> 
> Syncing to https://www.peeringdb.com/api
> 
> Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> 
> Fetching & updating all: org
> 
> Updates to be processed: 18439
> 
> Ignoring object updated after sync began: (org-6864)
> 
> Ignoring object updated after sync began: (org-13687)
> 
> Ignoring object updated after sync began: (org-23582)
> 
> Ignoring object updated after sync began: (org-26092)
> 
> Ignoring object updated after sync began: (org-26094)
> 
> Ignoring object updated after sync began: (org-26095)
> 
> Ignoring object updated after sync began: (org-26096)
> 
> Ignoring object updated after sync began: (org-26097)
> 
> Ignoring object updated after sync began: (org-26098)
> 
> Fetching & updating all: fac
> 
> Updates to be processed: 3714
> 
> Ignoring object updated after sync began: (fac-3634)
> 
> Ignoring object updated after sync began: (fac-8262)
> 
> Ignoring object updated after sync began: (fac-8582)
> 
> Ignoring object updated after sync began: (fac-8583)
> 
> Ignoring object updated after sync began: (fac-8584)
> 
> Ignoring object updated after sync began: (fac-8585)
> 
> Ignoring object updated after sync began: (fac-8586)
> 
> Ignoring object updated after sync began: (fac-8587)
> 
> Ignoring object updated after sync began: (fac-8588)
> 
> Ignoring object updated after sync began: (fac-8591)
> 
> Ignoring object updated after sync began: (fac-8592)
> 
> Ignoring object updated after sync began: (fac-8593)
> 
> Ignoring object updated after sync began: (fac-8594)
> 
> Ignoring object updated after sync began: (fac-8595)
> 
> Ignoring object updated after sync began: (fac-8597)
> 
> Ignoring object updated after sync began: (fac-8598)
> 
> Ignoring object updated after sync began: (fac-8599)
> 
> Ignoring object updated after sync began: (fac-8600)
> 
> Ignoring object updated after sync began: (fac-8601)
> 
> Ignoring object updated after sync began: (fac-8602)
> 
> Ignoring object updated after sync began: (fac-8603)
> 
> Fetching & updating all: net
> 
> Updates to be processed: 18845
> 
> Ignoring object updated after sync began: (net-2913)
> 
> Ignoring object updated after sync began: (net-3087)
> 
> Ignoring object updated after sync began: (net-4823)
> 
> Ignoring object updated after sync began: (net-5442)
> 
> Ignoring object updated after sync began: (net-7482)
> 
> Ignoring object updated after sync began: (net-9435)
> 
> Ignoring object updated after sync began: (net-13681)
> 
> Ignoring object updated after sync began: (net-14816)
> 
> Ignoring object updated after sync began: (net-19500)
> 
> Ignoring object updated after sync began: (net-20575)
> 
> Ignoring object updated after sync began: (net-21292)
> 
> Ignoring object updated after sync began: (net-21916)
> 
> Ignoring object updated after sync began: (net-23222)
> 
> Ignoring object updated after sync began: (net-23266)
> 
> Ignoring object updated after sync began: (net-23280)
> 
> Ignoring object updated after sync began: (net-23283)
> 
> Ignoring object updated after sync began: (net-23284)
> 
> Ignoring object updated after sync began: (net-23285)
> 
> Traceback (most recent call last):
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", 
> line 8, in 
> 
>     sys.exit(main())
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
>  line 62, in main
> 
>     return handler(config=cfg, **vars(options))
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 20, in _wrapped
> 
>     r = func(*a, **k)
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 219, in handle
> 
>     client.update_all(rs)
> 
>   File 
>

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Asbjørn Sloth Tønnesen 

Hi Brian,

On 5/15/20 5:43 PM, Brian Dickson wrote:
(This is kind of urgent for me, my goal is to get a snapshot of the data, so any alternative workaround would be helpful 
in the meantime.)


Alternatively you could use:
https://git.2e8.dk/peeringdb-simplesync/about/

--
Best regards
Asbjørn Sloth Tønnesen
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Brian Dickson 
On Fri, May 15, 2020 at 12:57 PM Brian Dickson <
brian.peter.dick...@gmail.com> wrote:

> The local database is empty. I'll make sure that's the case just to be
> safe.
>
> # peeringdb --version
>
> peeringdb 1.0.0
>
>
> # django-admin --version
>
> 2.2.12
>
>
> # pip freeze
>
> asgiref==3.2.7
>
> certifi==2020.4.5.1
>
> cfu==1.5.0
>
> chardet==3.0.4
>
> click==7.1.2
>
> Django==2.2.12
>
> django-countries==6.1.2
>
> django-handleref==0.5.0
>
> django-inet==0.4.0
>
> django-peeringdb==2.0.0
>
> future==0.18.2
>
> idna==2.9
>
> munge==1.0.0
>
> mysqlclient==1.4.4
>
> peeringdb==1.0.0
>
> PyMySQL==0.9.3
>
> pytz==2020.1
>
> PyYAML==5.3.1
>
> requests==2.23.0
>
> six==1.14.0
>
> sqlparse==0.3.1
>
> twentyc.rpc==0.4.0
>
> urllib3==1.25.9
>


Here is the result when running the sync on a clean empty local database:

bash-3.2# peeringdb drop-tables

bash-3.2# peeringdb sync

Syncing to https://www.peeringdb.com/api

Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc

Fetching & updating all: org

Updates to be processed: 18439

Ignoring object updated after sync began: (org-6864)

Ignoring object updated after sync began: (org-13687)

Ignoring object updated after sync began: (org-23582)

Ignoring object updated after sync began: (org-26092)

Ignoring object updated after sync began: (org-26094)

Ignoring object updated after sync began: (org-26095)

Ignoring object updated after sync began: (org-26096)

Ignoring object updated after sync began: (org-26097)

Ignoring object updated after sync began: (org-26098)

Fetching & updating all: fac

Updates to be processed: 3714

Ignoring object updated after sync began: (fac-3634)

Ignoring object updated after sync began: (fac-8262)

Ignoring object updated after sync began: (fac-8582)

Ignoring object updated after sync began: (fac-8583)

Ignoring object updated after sync began: (fac-8584)

Ignoring object updated after sync began: (fac-8585)

Ignoring object updated after sync began: (fac-8586)

Ignoring object updated after sync began: (fac-8587)

Ignoring object updated after sync began: (fac-8588)

Ignoring object updated after sync began: (fac-8591)

Ignoring object updated after sync began: (fac-8592)

Ignoring object updated after sync began: (fac-8593)

Ignoring object updated after sync began: (fac-8594)

Ignoring object updated after sync began: (fac-8595)

Ignoring object updated after sync began: (fac-8597)

Ignoring object updated after sync began: (fac-8598)

Ignoring object updated after sync began: (fac-8599)

Ignoring object updated after sync began: (fac-8600)

Ignoring object updated after sync began: (fac-8601)

Ignoring object updated after sync began: (fac-8602)

Ignoring object updated after sync began: (fac-8603)

Fetching & updating all: net

Updates to be processed: 18845

Ignoring object updated after sync began: (net-2913)

Ignoring object updated after sync began: (net-3087)

Ignoring object updated after sync began: (net-4823)

Ignoring object updated after sync began: (net-5442)

Ignoring object updated after sync began: (net-7482)

Ignoring object updated after sync began: (net-9435)

Ignoring object updated after sync began: (net-13681)

Ignoring object updated after sync began: (net-14816)

Ignoring object updated after sync began: (net-19500)

Ignoring object updated after sync began: (net-20575)

Ignoring object updated after sync began: (net-21292)

Ignoring object updated after sync began: (net-21916)

Ignoring object updated after sync began: (net-23222)

Ignoring object updated after sync began: (net-23266)

Ignoring object updated after sync began: (net-23280)

Ignoring object updated after sync began: (net-23283)

Ignoring object updated after sync began: (net-23284)

Ignoring object updated after sync began: (net-23285)

Traceback (most recent call last):

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
line 8, in 

sys.exit(main())

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
line 62, in main

return handler(config=cfg, **vars(options))

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
line 20, in _wrapped

r = func(*a, **k)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
line 219, in handle

client.update_all(rs)

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 66, in update_all

self._atomic_update(lambda: ctx.sync_resource(r, since=since))

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 78, in _atomic_update

sync_func()

  File
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
line 66, in 

self._atomic_update(lambda:

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Brian Dickson 
The local database is empty. I'll make sure that's the case just to be safe.

# peeringdb --version

peeringdb 1.0.0


# django-admin --version

2.2.12


# pip freeze

asgiref==3.2.7

certifi==2020.4.5.1

cfu==1.5.0

chardet==3.0.4

click==7.1.2

Django==2.2.12

django-countries==6.1.2

django-handleref==0.5.0

django-inet==0.4.0

django-peeringdb==2.0.0

future==0.18.2

idna==2.9

munge==1.0.0

mysqlclient==1.4.4

peeringdb==1.0.0

PyMySQL==0.9.3

pytz==2020.1

PyYAML==5.3.1

requests==2.23.0

six==1.14.0

sqlparse==0.3.1

twentyc.rpc==0.4.0

urllib3==1.25.9

On Fri, May 15, 2020 at 10:55 AM Chris Caputo  wrote:

> Hi.  Does restarting with a fresh/empty local database help at all?
>
> Also, what does the following show?:
>
> peeringdb --version
> django-admin --version
> pip freeze
>
> Thanks,
> Chris
>
> On Fri, 15 May 2020, Brian Dickson wrote:
> > Arnold Nipper said to send to this list.
> >
> > (This is kind of urgent for me, my goal is to get a snapshot of the
> data, so any alternative workaround would be helpful in the
> > meantime.)
> >
> > I'm in the process of setting up a local copy, and have run into a
> problem after getting the peeringdb-py stuff set up.
> >
> > When I do peeringdb sync (with the correct mysql database UTF8 stuff and
> everything,), I am currently getting a persistent error
> > which aborts the sync with no data in the tables:
> >
> > (Apologies in advance for the long error output text.)
> > Syncing to https://www.peeringdb.com/api
> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> > Fetching & updating all: org
> > Updates to be processed: 5
> > Ignoring object updated after sync began: (org-26015)
> > Ignoring object updated after sync began: (org-26052)
> > Ignoring object updated after sync began: (org-17918)
> > Ignoring object updated after sync began: (org-26053)
> > Fetching & updating all: fac
> > Updates to be processed: 0
> > Fetching & updating all: net
> > Updates to be processed: 18800
> > Ignoring object updated after sync began: (net-1356)
> > Ignoring object updated after sync began: (net-3684)
> > Ignoring object updated after sync began: (net-7924)
> > Ignoring object updated after sync began: (net-10733)
> > Ignoring object updated after sync began: (net-13084)
> > Ignoring object updated after sync began: (net-14581)
> > Ignoring object updated after sync began: (net-15702)
> > Traceback (most recent call last):
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> > line 84, in _execute
> > return self.cursor.execute(sql, params)
> >   File
> >
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
> line
> > 71, in execute
> > return self.cursor.execute(query, args)
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 209, in
> > execute
> > res = self._query(query)
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
> line 315, in
> > _query
> > db.query(q)
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
> line
> > 226, in query
> > _mysql.connection.query(self, query)
> > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child
> row: a foreign key constraint fails
> > (`peeringdb`.`peeringdb_network`, CONSTRAINT
> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
> > (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
> >
> > The above exception was the direct cause of the following exception:
> >
> > Traceback (most recent call last):
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
> line 8, in 
> > sys.exit(main())
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
> line 62, in
> > main
> > return handler(config=cfg, **vars(options))
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 20,
> > in _wrapped
> > r = func(*a, **k)
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
> line 219,
> > in handle
> > client.update_all(rs)
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 66,
> > in update_all
> > self._atomic_update(lambda: ctx.sync_resource(r, since=since))
> >   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
> line 78,
> > in _atomic_update
> > sync_func()
> >   File
>

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Chris Caputo 
Hi.  Does restarting with a fresh/empty local database help at all?

Also, what does the following show?:

peeringdb --version
django-admin --version
pip freeze

Thanks,
Chris

On Fri, 15 May 2020, Brian Dickson wrote:
> Arnold Nipper said to send to this list.
> 
> (This is kind of urgent for me, my goal is to get a snapshot of the data, so 
> any alternative workaround would be helpful in the
> meantime.)
> 
> I'm in the process of setting up a local copy, and have run into a problem 
> after getting the peeringdb-py stuff set up.
> 
> When I do peeringdb sync (with the correct mysql database UTF8 stuff and 
> everything,), I am currently getting a persistent error
> which aborts the sync with no data in the tables:
> 
> (Apologies in advance for the long error output text.)
> Syncing to https://www.peeringdb.com/api
> Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> Fetching & updating all: org
> Updates to be processed: 5
> Ignoring object updated after sync began: (org-26015)
> Ignoring object updated after sync began: (org-26052)
> Ignoring object updated after sync began: (org-17918)
> Ignoring object updated after sync began: (org-26053)
> Fetching & updating all: fac
> Updates to be processed: 0
> Fetching & updating all: net
> Updates to be processed: 18800
> Ignoring object updated after sync began: (net-1356)
> Ignoring object updated after sync began: (net-3684)
> Ignoring object updated after sync began: (net-7924)
> Ignoring object updated after sync began: (net-10733)
> Ignoring object updated after sync began: (net-13084)
> Ignoring object updated after sync began: (net-14581)
> Ignoring object updated after sync began: (net-15702)
> Traceback (most recent call last):
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>     return self.cursor.execute(sql, params)
>   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
>  line
> 71, in execute
>     return self.cursor.execute(query, args)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>  line 209, in
> execute
>     res = self._query(query)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>  line 315, in
> _query
>     db.query(q)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
>  line
> 226, in query
>     _mysql.connection.query(self, query)
> MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: 
> a foreign key constraint fails
> (`peeringdb`.`peeringdb_network`, CONSTRAINT 
> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
> (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", 
> line 8, in 
>     sys.exit(main())
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
>  line 62, in
> main
>     return handler(config=cfg, **vars(options))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 20,
> in _wrapped
>     r = func(*a, **k)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 219,
> in handle
>     client.update_all(rs)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66,
> in update_all
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 78,
> in _atomic_update
>     sync_func()
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66,
> in 
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line
> 65, in _wrapped
>     return loop.run_until_complete(func(*a, **k))
>   File 
> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
>  line 488, in
> run_until_complete
>     return future.result()
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line
> 41, in _wrapped
>     item = gen.send(r)
>   File 
>

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Brian Dickson 
Arnold Nipper said to send to this list.

(This is kind of urgent for me, my goal is to get a snapshot of the data,
so any alternative workaround would be helpful in the meantime.)

I'm in the process of setting up a local copy, and have run into a problem
after getting the peeringdb-py stuff set up.

When I do peeringdb sync (with the correct mysql database UTF8 stuff and
everything,), I am currently getting a persistent error which aborts the
sync with no data in the tables:

(Apologies in advance for the long error output text.)
Syncing to https://www.peeringdb.com/api
Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
Fetching & updating all: org
Updates to be processed: 5
Ignoring object updated after sync began: (org-26015)
Ignoring object updated after sync began: (org-26052)
Ignoring object updated after sync began: (org-17918)
Ignoring object updated after sync began: (org-26053)
Fetching & updating all: fac
Updates to be processed: 0
Fetching & updating all: net
Updates to be processed: 18800
Ignoring object updated after sync began: (net-1356)
Ignoring object updated after sync began: (net-3684)
Ignoring object updated after sync began: (net-7924)
Ignoring object updated after sync began: (net-10733)
Ignoring object updated after sync began: (net-13084)
Ignoring object updated after sync began: (net-14581)
Ignoring object updated after sync began: (net-15702)
Traceback (most recent call last):
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", line 84,
in _execute
return self.cursor.execute(sql, params)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line
71, in execute
return self.cursor.execute(query, args)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in
execute
res = self._query(query)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in
query
_mysql.connection.query(self, query)
MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child
row: a foreign key constraint fails (`peeringdb`.`peeringdb_network`,
CONSTRAINT `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id`
FOREIGN KEY (`org_id`) REFERENCES `peeringdb_organization` (`id`))')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File 
"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb",
line 8, in 
sys.exit(main())
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", line 62, in main
return handler(config=cfg, **vars(options))
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 20, in
_wrapped
r = func(*a, **k)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", line 219, in
handle
client.update_all(rs)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in
update_all
self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 78, in
_atomic_update
sync_func()
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 66, in

self._atomic_update(lambda: ctx.sync_resource(r, since=since))
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 65, in
_wrapped
return loop.run_until_complete(func(*a, **k))
  File "/opt/local/Library/Frameworks/Python.framework/
Versions/3.6/lib/python3.6/asyncio/base_events.py", line 488, in
run_until_complete
return future.result()
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", line 41, in
_wrapped
item = gen.send(r)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", line 274, in
sync_row
B.save(obj)
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py",
line 148, in save
obj.save()
  File "/Users/bdickson1/Documents/projects/site-selection/
pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", line 741, in
save
force_update=force_update, update_fields=update_fields)
  File