Hi Brian, Thanks for the feedback. I understand it is a specifi issue with packagecloud.io I am suprised to have a metadata file size issue with this repo, there are only 111 rpm packages and I have been synchronizing already some bigger repos without a problem :)
Regards Paul On Tue, Nov 25, 2014 at 6:10 PM, Brian Bouterse <[email protected]> wrote: > Hi Paul, > > Mongo does have a 16MB document limit [0], and this repo must have metadata > that is a little larger than that. I was able to reproduce this bug, and I > documented it here [1]. I did find some old BSON related Pulp bugs [2] [3], > but those all seemed to be resolved so [1] is the only open BSON related bug > I know of. > > [0]: http://docs.mongodb.org/manual/reference/limits/#bson-documents > [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1167950 > [2]: https://bugzilla.redhat.com/show_bug.cgi?id=748812 > [3]: https://bugzilla.redhat.com/show_bug.cgi?id=620495 > > -Brian > > > ----- Original Message ----- >> From: "Paul Gonin" <[email protected]> >> To: "Michael Hrivnak" <[email protected]>, [email protected] >> Sent: Monday, November 24, 2014 4:32:01 PM >> Subject: Re: [Pulp-list] Errors in pulp.log : BSON document too large >> >> Hi Michael, >> >> I ran manually the sync and it completed successfully from the admin >> client point of view: >> $ pulp-admin rpm repo sync run --repo-id=asis-chef-rhel6 >> +----------------------------------------------------------------------+ >> Synchronizing Repository [asis-chef-rhel6] >> +----------------------------------------------------------------------+ >> >> This command may be exited by pressing ctrl+c without affecting the actual >> operation on the server. >> >> Downloading metadata... >> [\] >> ... completed >> >> Downloading repository content... >> [== ] 5% >> RPMs: 1/5 items >> Delta RPMs: 0/0 items >> >> ... completed >> >> Downloading distribution files... >> [==================================================] 100% >> Distributions: 0/0 items >> ... completed >> >> Importing errata... >> [-] >> ... completed >> >> Importing package groups/categories... >> [-] >> ... completed >> >> Publishing packages... >> [==== ] 9% >> Packages: 5/54 items >> Publishing distributions... >> [==================================================] 100% >> Distributions: 0/0 items >> [==================================================] 100% >> Packages: 54/54 items >> ... completed >> >> Generating metadata >> [|] >> ... completed >> >> Publishing repository over HTTP >> [-] >> ... completed >> >> no error in ~/.pulp/admin.log >> >> In pulp.log : >> 2014-11-24 21:55:26,888 pulp.plugins.conduits.mixins:ERROR: Content >> unit association failed [Unit [key={'name': 'private-chef', >> 'checksum': '2d068c98c36c4eb548773efc616fec5c36bd2bdd', 'epoch': '0', >> 'version': '11.2.3', 'release': '1.el6', 'arch': 'x86_64', >> 'checksumtype': 'sha'}] [type=rpm] [id=None]] >> 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document >> too large (16967489 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> 2014-11-24 21:56:36,058 pulp.plugins.conduits.mixins:ERROR: Content >> unit association failed [Unit [key={'name': 'private-chef', >> 'checksum': 'd5d8bda7a3c260d42bbc79adb318cc05c794ae55', 'epoch': '0', >> 'version': '11.2.5', 'release': '1.el6', 'arch': 'x86_64', >> 'checksumtype': 'sha'}] [type=rpm] [id=None]] >> 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document >> too large (16968021 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> 2014-11-24 21:56:45,599 pulp.plugins.conduits.mixins:ERROR: Content >> unit association failed [Unit [key={'name': 'private-chef', >> 'checksum': '8ffc55a8eaaf5ce31d9a37aa257bbf6ffdd0aa34', 'epoch': '0', >> 'version': '11.2.2', 'release': '1.el6', 'arch': 'x86_64', >> 'checksumtype': 'sha'}] [type=rpm] [id=None]] >> 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document >> too large (16959923 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> 2014-11-24 21:57:03,051 pulp.plugins.conduits.mixins:ERROR: Content >> unit association failed [Unit [key={'name': 'private-chef', >> 'checksum': 'a0ef74235e3826dbf21ebea40070832d8430d988', 'epoch': '0', >> 'version': '11.2.4', 'release': '1.el6', 'arch': 'x86_64', >> 'checksumtype': 'sha'}] [type=rpm] [id=None]] >> 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document >> too large (16992417 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> >> In nectar.log >> 2014-11-24 21:50:52,988 >> requests.packages.urllib3.connectionpool:DEBUG: "GET >> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/904.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=PuNQ0kK68ejp4jkxhVgmWlgpHHY%3D&Expires=1416862551 >> HTTP/1.1" 200 576836991 >> 2014-11-24 21:50:53,197 >> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout >> to <object object at 0x7fe99d14c100> >> 2014-11-24 21:50:53,310 >> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout >> to <object object at 0x7fe99d14c100> >> 2014-11-24 21:50:53,430 >> requests.packages.urllib3.connectionpool:DEBUG: "GET >> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1290.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=AUyaEOnkxhQ7nRAHeGDuxsFLZJE%3D&Expires=1416862553 >> HTTP/1.1" 200 577050972 >> 2014-11-24 21:50:53,550 >> requests.packages.urllib3.connectionpool:DEBUG: "GET >> https://packagecloud.io/chef/stable/el/6/x86_64/private-chef-11.2.5-1.el6.x86_64.rpm >> HTTP/1.1" 302 0 >> 2014-11-24 21:50:53,552 requests.packages.urllib3.connectionpool:INFO: >> Starting new HTTPS connection (1): >> packagecloud-repositories.s3.amazonaws.com >> 2014-11-24 21:50:53,758 >> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout >> to <object object at 0x7fe99d14c100> >> 2014-11-24 21:50:53,998 >> requests.packages.urllib3.connectionpool:DEBUG: "GET >> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1366.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=bgbKpTUhe3bNNWJHTgLJDsTcJZ0%3D&Expires=1416862552 >> HTTP/1.1" 200 578311972 >> 2014-11-24 21:50:54,248 >> requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout >> to <object object at 0x7fe99d14c100> >> 2014-11-24 21:50:54,478 >> requests.packages.urllib3.connectionpool:DEBUG: "GET >> https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1448.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=vGIQTaz8OYV8cIrUCQY4dsxvQ6U%3D&Expires=1416862553 >> HTTP/1.1" 200 577030942 >> 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document >> too large (16967489 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", >> line 110, in _fire_event_to_listener >> event_listener_callback(*args, **kwargs) >> File >> >> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", >> line 97, in download_succeeded >> self.sync_conduit.save_unit(unit) >> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", >> line 485, in save_unit >> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) >> File >> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", >> line 47, in add_content_unit >> collection.insert(unit_doc, safe=True) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 114, in retry >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", >> line 357, in insert >> continue_on_error, self.__uuid_subtype), safe) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 143, in _with_end_request >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 910, in _send_message >> (request_id, data) = self.__check_bson_size(message) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 881, in __check_bson_size >> (max_doc_size, self.__max_bson_size)) >> ImporterConduitException: BSON document too large (16967489 bytes) - >> the connected server supports BSON document sizes up to 16777216 >> bytes. >> 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document >> too large (16968021 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", >> line 110, in _fire_event_to_listener >> event_listener_callback(*args, **kwargs) >> File >> >> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", >> line 97, in download_succeeded >> self.sync_conduit.save_unit(unit) >> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", >> line 485, in save_unit >> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) >> File >> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", >> line 47, in add_content_unit >> collection.insert(unit_doc, safe=True) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 114, in retry >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", >> line 357, in insert >> continue_on_error, self.__uuid_subtype), safe) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 143, in _with_end_request >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 910, in _send_message >> (request_id, data) = self.__check_bson_size(message) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 881, in __check_bson_size >> (max_doc_size, self.__max_bson_size)) >> ImporterConduitException: BSON document too large (16968021 bytes) - >> the connected server supports BSON document sizes up to 16777216 >> bytes. >> 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document >> too large (16959923 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", >> line 110, in _fire_event_to_listener >> event_listener_callback(*args, **kwargs) >> File >> >> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", >> line 97, in download_succeeded >> self.sync_conduit.save_unit(unit) >> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", >> line 485, in save_unit >> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) >> File >> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", >> line 47, in add_content_unit >> collection.insert(unit_doc, safe=True) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 114, in retry >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", >> line 357, in insert >> continue_on_error, self.__uuid_subtype), safe) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 143, in _with_end_request >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 910, in _send_message >> (request_id, data) = self.__check_bson_size(message) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 881, in __check_bson_size >> (max_doc_size, self.__max_bson_size)) >> ImporterConduitException: BSON document too large (16959923 bytes) - >> the connected server supports BSON document sizes up to 16777216 >> bytes. >> 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document >> too large (16992417 bytes) - the connected server supports BSON >> document sizes up to 16777216 bytes. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", >> line 110, in _fire_event_to_listener >> event_listener_callback(*args, **kwargs) >> File >> >> "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", >> line 97, in download_succeeded >> self.sync_conduit.save_unit(unit) >> File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", >> line 485, in save_unit >> unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) >> File >> "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", >> line 47, in add_content_unit >> collection.insert(unit_doc, safe=True) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 114, in retry >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", >> line 357, in insert >> continue_on_error, self.__uuid_subtype), safe) >> File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", >> line 143, in _with_end_request >> return method(*args, **kwargs) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 910, in _send_message >> (request_id, data) = self.__check_bson_size(message) >> File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", >> line 881, in __check_bson_size >> (max_doc_size, self.__max_bson_size)) >> ImporterConduitException: BSON document too large (16992417 bytes) - >> the connected server supports BSON document sizes up to 16777216 >> bytes. >> 2014-11-24 21:57:03,062 nectar.downloaders.threaded:DEBUG: starting workers >> >> >> mongodb.log empty >> >> Thanks ! >> Regards >> Paul >> >> On Mon, Nov 24, 2014 at 7:44 PM, Michael Hrivnak <[email protected]> wrote: >> > Paul, >> > >> > Is there a full stack trace in the log that you can share? >> > >> > Thanks, >> > Michael >> > >> > ----- Original Message ----- >> > From: "Paul Gonin" <[email protected]> >> > To: [email protected] >> > Sent: Monday, November 24, 2014 5:36:30 AM >> > Subject: [Pulp-list] Errors in pulp.log : BSON document too large >> > >> > Hi ! >> > >> > A few days ago I started synchronizing chef repo from packagecloud.io >> > https://packagecloud.io/chef/stable/el/6/x86_64 >> > >> > I scheduled the sync to be ran every night at 2am. >> > >> > Since then I have the following MongoDB related error in pulp.log >> > which seems related to MongoDB document size limit to 16MB >> > >> > 2014-11-22 02:36:56,824 nectar.downloaders.base:ERROR: BSON document >> > too large (16968021 bytes) - the connected server supports BSON >> > document sizes up to 16777216 bytes. >> > >> > Any idea how to address it ? >> > I looked in Bugzilla if it is known issue but could not anything >> > related to BSON. >> > >> > Thanks >> > Paul >> > >> > _______________________________________________ >> > Pulp-list mailing list >> > [email protected] >> > https://www.redhat.com/mailman/listinfo/pulp-list >> >> _______________________________________________ >> Pulp-list mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/pulp-list >> _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
