Hi Paul, Mongo does have a 16MB document limit [0], and this repo must have metadata that is a little larger than that. I was able to reproduce this bug, and I documented it here [1]. I did find some old BSON related Pulp bugs [2] [3], but those all seemed to be resolved so [1] is the only open BSON related bug I know of.
[0]: http://docs.mongodb.org/manual/reference/limits/#bson-documents [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1167950 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=748812 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=620495 -Brian ----- Original Message ----- > From: "Paul Gonin" <[email protected]> > To: "Michael Hrivnak" <[email protected]>, [email protected] > Sent: Monday, November 24, 2014 4:32:01 PM > Subject: Re: [Pulp-list] Errors in pulp.log : BSON document too large > > Hi Michael, > > I ran manually the sync and it completed successfully from the admin > client point of view: > $ pulp-admin rpm repo sync run --repo-id=asis-chef-rhel6 > +----------------------------------------------------------------------+ > Synchronizing Repository [asis-chef-rhel6] > +----------------------------------------------------------------------+ > > This command may be exited by pressing ctrl+c without affecting the actual > operation on the server. > > Downloading metadata... > [\] > ... completed > > Downloading repository content... > [== ] 5% > RPMs: 1/5 items > Delta RPMs: 0/0 items > > ... completed > > Downloading distribution files... > [==================================================] 100% > Distributions: 0/0 items > ... completed > > Importing errata... > [-] > ... completed > > Importing package groups/categories... > [-] > ... completed > > Publishing packages... > [==== ] 9% > Packages: 5/54 items > Publishing distributions... > [==================================================] 100% > Distributions: 0/0 items > [==================================================] 100% > Packages: 54/54 items > ... completed > > Generating metadata > [|] > ... completed > > Publishing repository over HTTP > [-] > ... completed > > no error in ~/.pulp/admin.log > > In pulp.log : > 2014-11-24 21:55:26,888 pulp.plugins.conduits.mixins:ERROR: Content > unit association failed [Unit [key={'name': 'private-chef', > 'checksum': '2d068c98c36c4eb548773efc616fec5c36bd2bdd', 'epoch': '0', > 'version': '11.2.3', 'release': '1.el6', 'arch': 'x86_64', > 'checksumtype': 'sha'}] [type=rpm] [id=None]] > 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document > too large (16967489 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > 2014-11-24 21:56:36,058 pulp.plugins.conduits.mixins:ERROR: Content > unit association failed [Unit [key={'name': 'private-chef', > 'checksum': 'd5d8bda7a3c260d42bbc79adb318cc05c794ae55', 'epoch': '0', > 'version': '11.2.5', 'release': '1.el6', 'arch': 'x86_64', > 'checksumtype': 'sha'}] [type=rpm] [id=None]] > 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document > too large (16968021 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > 2014-11-24 21:56:45,599 pulp.plugins.conduits.mixins:ERROR: Content > unit association failed [Unit [key={'name': 'private-chef', > 'checksum': '8ffc55a8eaaf5ce31d9a37aa257bbf6ffdd0aa34', 'epoch': '0', > 'version': '11.2.2', 'release': '1.el6', 'arch': 'x86_64', > 'checksumtype': 'sha'}] [type=rpm] [id=None]] > 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document > too large (16959923 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > 2014-11-24 21:57:03,051 pulp.plugins.conduits.mixins:ERROR: Content > unit association failed [Unit [key={'name': 'private-chef', > 'checksum': 'a0ef74235e3826dbf21ebea40070832d8430d988', 'epoch': '0', > 'version': '11.2.4', 'release': '1.el6', 'arch': 'x86_64', > 'checksumtype': 'sha'}] [type=rpm] [id=None]] > 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document > too large (16992417 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > > In nectar.log > 2014-11-24 21:50:52,988 > requests.packages.urllib3.connectionpool:DEBUG: "GET > https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/904.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=PuNQ0kK68ejp4jkxhVgmWlgpHHY%3D&Expires=1416862551 > HTTP/1.1" 200 576836991 > 2014-11-24 21:50:53,197 > requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout > to <object object at 0x7fe99d14c100> > 2014-11-24 21:50:53,310 > requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout > to <object object at 0x7fe99d14c100> > 2014-11-24 21:50:53,430 > requests.packages.urllib3.connectionpool:DEBUG: "GET > https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1290.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=AUyaEOnkxhQ7nRAHeGDuxsFLZJE%3D&Expires=1416862553 > HTTP/1.1" 200 577050972 > 2014-11-24 21:50:53,550 > requests.packages.urllib3.connectionpool:DEBUG: "GET > https://packagecloud.io/chef/stable/el/6/x86_64/private-chef-11.2.5-1.el6.x86_64.rpm > HTTP/1.1" 302 0 > 2014-11-24 21:50:53,552 requests.packages.urllib3.connectionpool:INFO: > Starting new HTTPS connection (1): > packagecloud-repositories.s3.amazonaws.com > 2014-11-24 21:50:53,758 > requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout > to <object object at 0x7fe99d14c100> > 2014-11-24 21:50:53,998 > requests.packages.urllib3.connectionpool:DEBUG: "GET > https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1366.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=bgbKpTUhe3bNNWJHTgLJDsTcJZ0%3D&Expires=1416862552 > HTTP/1.1" 200 578311972 > 2014-11-24 21:50:54,248 > requests.packages.urllib3.connectionpool:DEBUG: Setting read timeout > to <object object at 0x7fe99d14c100> > 2014-11-24 21:50:54,478 > requests.packages.urllib3.connectionpool:DEBUG: "GET > https://packagecloud-repositories.s3.amazonaws.com/141/315/el/6/package_files/1448.rpm?AWSAccessKeyId=AKIAI44QGWC7C5WEV4XA&Signature=vGIQTaz8OYV8cIrUCQY4dsxvQ6U%3D&Expires=1416862553 > HTTP/1.1" 200 577030942 > 2014-11-24 21:55:26,890 nectar.downloaders.base:ERROR: BSON document > too large (16967489 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", > line 110, in _fire_event_to_listener > event_listener_callback(*args, **kwargs) > File > > "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", > line 97, in download_succeeded > self.sync_conduit.save_unit(unit) > File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", > line 485, in save_unit > unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) > File > "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", > line 47, in add_content_unit > collection.insert(unit_doc, safe=True) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 114, in retry > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", > line 357, in insert > continue_on_error, self.__uuid_subtype), safe) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 143, in _with_end_request > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 910, in _send_message > (request_id, data) = self.__check_bson_size(message) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 881, in __check_bson_size > (max_doc_size, self.__max_bson_size)) > ImporterConduitException: BSON document too large (16967489 bytes) - > the connected server supports BSON document sizes up to 16777216 > bytes. > 2014-11-24 21:56:36,059 nectar.downloaders.base:ERROR: BSON document > too large (16968021 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", > line 110, in _fire_event_to_listener > event_listener_callback(*args, **kwargs) > File > > "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", > line 97, in download_succeeded > self.sync_conduit.save_unit(unit) > File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", > line 485, in save_unit > unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) > File > "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", > line 47, in add_content_unit > collection.insert(unit_doc, safe=True) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 114, in retry > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", > line 357, in insert > continue_on_error, self.__uuid_subtype), safe) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 143, in _with_end_request > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 910, in _send_message > (request_id, data) = self.__check_bson_size(message) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 881, in __check_bson_size > (max_doc_size, self.__max_bson_size)) > ImporterConduitException: BSON document too large (16968021 bytes) - > the connected server supports BSON document sizes up to 16777216 > bytes. > 2014-11-24 21:56:45,600 nectar.downloaders.base:ERROR: BSON document > too large (16959923 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", > line 110, in _fire_event_to_listener > event_listener_callback(*args, **kwargs) > File > > "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", > line 97, in download_succeeded > self.sync_conduit.save_unit(unit) > File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", > line 485, in save_unit > unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) > File > "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", > line 47, in add_content_unit > collection.insert(unit_doc, safe=True) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 114, in retry > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", > line 357, in insert > continue_on_error, self.__uuid_subtype), safe) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 143, in _with_end_request > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 910, in _send_message > (request_id, data) = self.__check_bson_size(message) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 881, in __check_bson_size > (max_doc_size, self.__max_bson_size)) > ImporterConduitException: BSON document too large (16959923 bytes) - > the connected server supports BSON document sizes up to 16777216 > bytes. > 2014-11-24 21:57:03,052 nectar.downloaders.base:ERROR: BSON document > too large (16992417 bytes) - the connected server supports BSON > document sizes up to 16777216 bytes. > Traceback (most recent call last): > File "/usr/lib/python2.6/site-packages/nectar/downloaders/base.py", > line 110, in _fire_event_to_listener > event_listener_callback(*args, **kwargs) > File > > "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/listener.py", > line 97, in download_succeeded > self.sync_conduit.save_unit(unit) > File "/usr/lib/python2.6/site-packages/pulp/plugins/conduits/mixins.py", > line 485, in save_unit > unit.id = content_manager.add_content_unit(unit.type_id, None, pulp_unit) > File > "/usr/lib/python2.6/site-packages/pulp/server/managers/content/cud.py", > line 47, in add_content_unit > collection.insert(unit_doc, safe=True) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 114, in retry > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", > line 357, in insert > continue_on_error, self.__uuid_subtype), safe) > File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", > line 143, in _with_end_request > return method(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 910, in _send_message > (request_id, data) = self.__check_bson_size(message) > File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", > line 881, in __check_bson_size > (max_doc_size, self.__max_bson_size)) > ImporterConduitException: BSON document too large (16992417 bytes) - > the connected server supports BSON document sizes up to 16777216 > bytes. > 2014-11-24 21:57:03,062 nectar.downloaders.threaded:DEBUG: starting workers > > > mongodb.log empty > > Thanks ! > Regards > Paul > > On Mon, Nov 24, 2014 at 7:44 PM, Michael Hrivnak <[email protected]> wrote: > > Paul, > > > > Is there a full stack trace in the log that you can share? > > > > Thanks, > > Michael > > > > ----- Original Message ----- > > From: "Paul Gonin" <[email protected]> > > To: [email protected] > > Sent: Monday, November 24, 2014 5:36:30 AM > > Subject: [Pulp-list] Errors in pulp.log : BSON document too large > > > > Hi ! > > > > A few days ago I started synchronizing chef repo from packagecloud.io > > https://packagecloud.io/chef/stable/el/6/x86_64 > > > > I scheduled the sync to be ran every night at 2am. > > > > Since then I have the following MongoDB related error in pulp.log > > which seems related to MongoDB document size limit to 16MB > > > > 2014-11-22 02:36:56,824 nectar.downloaders.base:ERROR: BSON document > > too large (16968021 bytes) - the connected server supports BSON > > document sizes up to 16777216 bytes. > > > > Any idea how to address it ? > > I looked in Bugzilla if it is known issue but could not anything > > related to BSON. > > > > Thanks > > Paul > > > > _______________________________________________ > > Pulp-list mailing list > > [email protected] > > https://www.redhat.com/mailman/listinfo/pulp-list > > _______________________________________________ > Pulp-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/pulp-list > _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
