On 08/02/2010 11:00 PM, Mike McCune wrote:
I spent some time today tracking down the exception that a few people
were seeing:
InvalidDocument: document too large - BSON documents are limited to
4 MB
I wrote a unit test to create a repo and add 5,000 packages to the
repository and easily hit the above error:
File "/home/mmccune/devel/pulp/test/unit/test_api.py", line 615, in
test_sync_large_repo
self.rapi.update(repo)
File
"/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line
166, in _audit
_record_event()
File
"/home/mmccune/devel/pulp/test/unit/../../src/pulp/auditing.py", line
141, in _record_event
_objdb.insert(event, safe=False, check_keys=False)
File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 245,
in insert
message.insert(self.__full_name, docs, check_keys, safe), safe)
InvalidDocument: document too large - BSON documents are limited to 4 MB
Note the trace is contained within the auditing insert() call.
Turns out the Event class has both a 'params' field and a field called
'action' but the action field also contained a copy of the params so
for a large Repo document you get an Event document that is 2-3X the
size of the actual parameter being audited.
Jason is working to trim this down but the short story is, we aren't
hitting the 4MB limit because of the structure of our actual Repo
document, we are hitting it because of the way we were auditing the
parameters of a method call.
Jason also in his latest update took out the repo from the param
logging in RepoApi.update()
@audit(params=[])
def update(self, repo_data):
Also, to make people feel more confortable, with the fix to the
audit() call I was able to create a repo with 20,000 packages without
error whereas before it would bail with a repo with 5,000.
Ah nice find Mike. Thanks for tracking this down.
~ Prad
_______________________________________________
Pulp-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/pulp-list
