I encounted a new issue that escaped my notice by some weird coincidence.
It turns that you use the pulp_settings section of the playbook, aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead of AWS_DEFAULT_ACL = None, which results in the cryptic "An error occurred (InvalidArgument) when calling the PutObject operation: None" Is there a way to ensure that Ansible converts aws_default_acl: None to AWS_DEFAULT_ACL = None? I tried setting it to null but I got an error. Cheers, Joey Dumont Technical Advisor, Knowledge, Information, and Technology Services National Research Council Canada / Governement of Canada [email protected]<mailto:[email protected]> / Tel: 613-990-8152 / Cell: 438-340-7436 Conseiller technique, Services du savoir, de l'information et de la technologie Conseil national de recherches Canada / Gouvernement du Canada [email protected]<mailto:[email protected]> / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436 ________________________________ From: [email protected] <[email protected]> on behalf of Dumont, Joey <[email protected]> Sent: 06 November 2019 11:36 To: Mike DePaulo Cc: [email protected] Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3) PR, as suggested: https://github.com/pulp/pulpcore/pull/366. Cheers, Joey Dumont Technical Advisor, Knowledge, Information, and Technology Services National Research Council Canada / Governement of Canada [email protected]<mailto:[email protected]> / Tel: 613-990-8152 / Cell: 438-340-7436 Conseiller technique, Services du savoir, de l'information et de la technologie Conseil national de recherches Canada / Gouvernement du Canada [email protected]<mailto:[email protected]> / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436 ________________________________ From: Mike DePaulo <[email protected]> Sent: 06 November 2019 10:52 To: Dumont, Joey Cc: David Davis; [email protected] Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3) Hi Joey, It sounds like aws_default_acl should be documented here then: https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws Care to submit a documentation PR? https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst Thanks, -Mike On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey <[email protected]<mailto:[email protected]>> wrote: Turns out the issue was on my end. I had to add aws_default_acl: None to the pulp_settings section of the playbook. The public-read ACL was incompatible with the BlockPublicAccess settings that I had set on my S3 bucket. I'm now encountering a different issue, but I'll start another thread for that one. Thanks for the pointers, they were very helpful! Joey Dumont Technical Advisor, Knowledge, Information, and Technology Services National Research Council Canada / Governement of Canada [email protected]<mailto:[email protected]> / Tel: 613-990-8152 / Cell: 438-340-7436 Conseiller technique, Services du savoir, de l'information et de la technologie Conseil national de recherches Canada / Gouvernement du Canada [email protected]<mailto:[email protected]> / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436 ________________________________ From: David Davis <[email protected]<mailto:[email protected]>> Sent: 01 November 2019 15:51 To: Dumont, Joey Cc: [email protected]<mailto:[email protected]> Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3) Unfortunately I don't know of a good way to debug the problem other than to dig into the code. If you want to debug from the Pulp code, you could stick a debugger in the artifact saver stage: https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200 What I would probably do though is stick a debug statement here in django-storages to see what params it's passing to boto3: https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511 You can see the location of django-storages with `pip show django-storages`. Sorry I don't have a better answer for you. Perhaps this is something we can improve in the future. Also, I'd be curious as to what the issue is as it sounds like everything should work in theory. David On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey <[email protected]<mailto:[email protected]>> wrote: I've installed the latest pulp3 using the Ansible installer using the following playbook: --- - hosts: mirrors vars: prereq_pip_packages: - django-storages - boto3 pulp_use_system_wide_pkgs: True pulp_default_admin_password: !vault | $ANSIBLE_VAULT;1.1;AES256 ... pulp_settings: secret_key: !vault | $ANSIBLE_VAULT;1.1;AES256 ... default_file_storage: 'storages.backends.s3boto3.S3Boto3Storage' aws_storage_bucket_name: 'xxx-pulp-storage' aws_s3_region_name: 'ca-central-1' aws_s3_addressing_style: "path" media_root: '/pulp3/' pulp_install_plugins: pulp-file: {} pulp-rpm: prereq_role: "pulp.pulp_rpm_prerequisites" # pulp-docker: {} roles: - pulp-database - pulp-workers - pulp-resource-manager - pulp-webserver - pulp-content environment: DJANGO_SETTINGS_MODULE: pulpcore.app.settings I also set up an RPM repo that uses S3 for storage. However, when I try to sync, I get an AccessDenied error. I know the instance profile is correct, as I can upload objects from that instance using both the AWS CLI and Boto3 without specifying credentials. How can I debug this further? Is there a way for me know what parameters are passed to the put_object boto3 call by the sync task? Cheers, Joey Dumont Technical Advisor, Knowledge, Information, and Technology Services National Research Council Canada / Governement of Canada [email protected]<mailto:[email protected]> / Tel: 613-990-8152 / Cell: 438-340-7436 Conseiller technique, Services du savoir, de l'information et de la technologie Conseil national de recherches Canada / Gouvernement du Canada [email protected]<mailto:[email protected]> / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436 _______________________________________________ Pulp-list mailing list [email protected]<mailto:[email protected]> https://www.redhat.com/mailman/listinfo/pulp-list _______________________________________________ Pulp-list mailing list [email protected]<mailto:[email protected]> https://www.redhat.com/mailman/listinfo/pulp-list -- Mike DePaulo He / Him / His Service Reliability Engineer, Pulp Red Hat<https://www.redhat.com/> IM: mikedep333 GPG: 51745404 [https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png]<https://www.redhat.com/>
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
