date:20191109

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

2019-11-09 Thread David Davis

You should be able to set aws_default_acl to null (ie "aws_default_acl: ~"
or "aws_default_acl: null"). Sounds like that's not working though. What
error did you get? Would you mind filing an issue?

https://pulp.plan.io/projects/pulp/issues/new

Thanks.

David


On Fri, Nov 8, 2019 at 10:20 PM Dumont, Joey 
wrote:

> I encounted a new issue that escaped my notice by some weird coincidence.
>
>
> It turns that you use the pulp_settings section of the playbook,
> aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead
> of AWS_DEFAULT_ACL = None, which results in the cryptic
>
>
> "An error occurred (InvalidArgument) when calling the PutObject operation:
> None"
>
> Is there a way to ensure that Ansible converts aws_default_acl: None to
> AWS_DEFAULT_ACL = None?
>
> I tried setting it to null but I got an error.
>
> Cheers,
> 
>
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> --
> *From:* pulp-list-boun...@redhat.com  on
> behalf of Dumont, Joey 
> *Sent:* 06 November 2019 11:36
> *To:* Mike DePaulo
> *Cc:* pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
>
> PR, as suggested: https://github.com/pulp/pulpcore/pull/366.
>
>
> Cheers,
>
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> --
> *From:* Mike DePaulo 
> *Sent:* 06 November 2019 10:52
> *To:* Dumont, Joey
> *Cc:* David Davis; pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
> Hi Joey,
>
> It sounds like aws_default_acl should be documented here then:
>
> https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws
>
> Care to submit a documentation PR?
> https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst
>
> Thanks,
> -Mike
>
> On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey 
> wrote:
>
>> Turns out the issue was on my end. I had to add aws_default_acl: None to
>> the pulp_settings section of the playbook. The public-read ACL was
>> incompatible with the BlockPublicAccess settings that I had set on my S3
>> bucket. 
>>
>> I'm now encountering a different issue, but I'll start another thread for
>> that one.
>>
>> Thanks for the pointers, they were very helpful!
>>
>> Joey Dumont
>>
>> Technical Advisor, Knowledge, Information, and Technology Services
>> National Research Council Canada / Governement of Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>
>> Conseiller technique, Services du savoir, de l'information et de la
>> technologie
>> Conseil national de recherches Canada / Gouvernement du Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>> 438-340-7436
>> --
>> *From:* David Davis 
>> *Sent:* 01 November 2019 15:51
>> *To:* Dumont, Joey
>> *Cc:* pulp-list@redhat.com
>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>> pulp on Amazon EC2 (pulp3)
>>
>> Unfortunately I don't know of a good way to debug the problem other than
>> to dig into the code. If you want to debug from the Pulp code, you could
>> stick a debugger in the artifact saver stage:
>>
>>
>> https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200
>>
>> What I would probably do though is stick a debug statement here in
>> django-storages to see what params it's passing to boto3:
>>
>>
>> https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511
>>
>> You can see the location of django-storages with `pip show
>> django-storages`.
>>
>> Sorry I don't have a better answer for you. Perhaps this is something we
>> can improve in the future. Also, I'd be curious as to what the issue is as
>> it sounds like everything should work in theory.
>>
>> David
>>
>>
>> On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey 
>> wrote:
>>
>>> I've installed the latest pulp3 using the Ansible installer using the
>>> following playbook:
>>>
>>>
>>> ---
>>> - hosts: mirrors
>>>   vars:
>>> prereq_pip_packages:
>>>   -

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

2019-11-09 Thread Mike DePaulo

I think that with Ansible syntax, you need to specify:
  AWS_DEFAULT_ACL:

I am fairly certain the lack of any value at the end would get interpreted
as Python None.

- Mike

On Sat, Nov 9, 2019 at 6:57 AM David Davis  wrote:

> You should be able to set aws_default_acl to null (ie "aws_default_acl: ~"
> or "aws_default_acl: null"). Sounds like that's not working though. What
> error did you get? Would you mind filing an issue?
>
> https://pulp.plan.io/projects/pulp/issues/new
>
> Thanks.
>
> David
>
>
> On Fri, Nov 8, 2019 at 10:20 PM Dumont, Joey 
> wrote:
>
>> I encounted a new issue that escaped my notice by some weird coincidence.
>>
>>
>> It turns that you use the pulp_settings section of the playbook,
>> aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead
>> of AWS_DEFAULT_ACL = None, which results in the cryptic
>>
>>
>> "An error occurred (InvalidArgument) when calling the PutObject
>> operation: None"
>>
>> Is there a way to ensure that Ansible converts aws_default_acl: None to
>> AWS_DEFAULT_ACL = None?
>>
>> I tried setting it to null but I got an error.
>>
>> Cheers,
>> 
>>
>>
>> Joey Dumont
>>
>> Technical Advisor, Knowledge, Information, and Technology Services
>> National Research Council Canada / Governement of Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>
>> Conseiller technique, Services du savoir, de l'information et de la
>> technologie
>> Conseil national de recherches Canada / Gouvernement du Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>> 438-340-7436
>> --
>> *From:* pulp-list-boun...@redhat.com  on
>> behalf of Dumont, Joey 
>> *Sent:* 06 November 2019 11:36
>> *To:* Mike DePaulo
>> *Cc:* pulp-list@redhat.com
>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>> pulp on Amazon EC2 (pulp3)
>>
>>
>> PR, as suggested: https://github.com/pulp/pulpcore/pull/366.
>>
>>
>> Cheers,
>>
>>
>> Joey Dumont
>>
>> Technical Advisor, Knowledge, Information, and Technology Services
>> National Research Council Canada / Governement of Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>
>> Conseiller technique, Services du savoir, de l'information et de la
>> technologie
>> Conseil national de recherches Canada / Gouvernement du Canada
>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>> 438-340-7436
>> --
>> *From:* Mike DePaulo 
>> *Sent:* 06 November 2019 10:52
>> *To:* Dumont, Joey
>> *Cc:* David Davis; pulp-list@redhat.com
>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>> pulp on Amazon EC2 (pulp3)
>>
>> Hi Joey,
>>
>> It sounds like aws_default_acl should be documented here then:
>>
>> https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws
>>
>> Care to submit a documentation PR?
>> https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst
>>
>> Thanks,
>> -Mike
>>
>> On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey 
>> wrote:
>>
>>> Turns out the issue was on my end. I had to add aws_default_acl: None to
>>> the pulp_settings section of the playbook. The public-read ACL was
>>> incompatible with the BlockPublicAccess settings that I had set on my S3
>>> bucket. 
>>>
>>> I'm now encountering a different issue, but I'll start another thread
>>> for that one.
>>>
>>> Thanks for the pointers, they were very helpful!
>>>
>>> Joey Dumont
>>>
>>> Technical Advisor, Knowledge, Information, and Technology Services
>>> National Research Council Canada / Governement of Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>
>>> Conseiller technique, Services du savoir, de l'information et de la
>>> technologie
>>> Conseil national de recherches Canada / Gouvernement du Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>> 438-340-7436
>>> --
>>> *From:* David Davis 
>>> *Sent:* 01 November 2019 15:51
>>> *To:* Dumont, Joey
>>> *Cc:* pulp-list@redhat.com
>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>> pulp on Amazon EC2 (pulp3)
>>>
>>> Unfortunately I don't know of a good way to debug the problem other than
>>> to dig into the code. If you want to debug from the Pulp code, you could
>>> stick a debugger in the artifact saver stage:
>>>
>>>
>>> https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200
>>>
>>> What I would probably do though is stick a debug statement here in
>>> django-storages to see what params it's passing to boto3:
>>>
>>>
>>> https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511
>>>
>>> You can see the location of django-storages with `pip show
>>> django-storages`.
>>>
>>> Sorry I don't have a better answer for you. Perhaps this is something we
>>> can improve in

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

2019-11-09 Thread Dumont, Joey

I have tried '~', 'null' and '' (without the quotes), and they all get rendered 
as null. This leads to an issue with running the migrations when running the 
Ansible script.


I opened an issue: https://pulp.plan.io/issues/5687


Maybe this should have gone in the pulp-ansible project?


Cheers,


Joey Dumont

Technical Advisor, Knowledge, Information, and Technology Services
National Research Council Canada / Governement of Canada
joey.dum...@nrc-cnrc.gc.ca / Tel: 
613-990-8152 / Cell: 438-340-7436

Conseiller technique, Services du savoir, de l'information et de la technologie
Conseil national de recherches Canada / Gouvernement du Canada
joey.dum...@nrc-cnrc.gc.ca / Tél.: 
613-990-8152 / Tél. cell.: 438-340-7436

From: Mike DePaulo 
Sent: 09 November 2019 13:47
To: David Davis
Cc: Dumont, Joey; pulp-list@redhat.com
Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on 
Amazon EC2 (pulp3)

I think that with Ansible syntax, you need to specify:
  AWS_DEFAULT_ACL:

I am fairly certain the lack of any value at the end would get interpreted as 
Python None.

- Mike

On Sat, Nov 9, 2019 at 6:57 AM David Davis 
mailto:davidda...@redhat.com>> wrote:
You should be able to set aws_default_acl to null (ie "aws_default_acl: ~" or 
"aws_default_acl: null"). Sounds like that's not working though. What error did 
you get? Would you mind filing an issue?

https://pulp.plan.io/projects/pulp/issues/new

Thanks.

David


On Fri, Nov 8, 2019 at 10:20 PM Dumont, Joey 
mailto:joey.dum...@nrc-cnrc.gc.ca>> wrote:

I encounted a new issue that escaped my notice by some weird coincidence.


It turns that you use the pulp_settings section of the playbook, 
aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead of 
AWS_DEFAULT_ACL = None, which results in the cryptic


"An error occurred (InvalidArgument) when calling the PutObject operation: None"

Is there a way to ensure that Ansible converts aws_default_acl: None to 
AWS_DEFAULT_ACL = None?

I tried setting it to null but I got an error.

Cheers,



Joey Dumont

Technical Advisor, Knowledge, Information, and Technology Services
National Research Council Canada / Governement of Canada
joey.dum...@nrc-cnrc.gc.ca / Tel: 
613-990-8152 / Cell: 438-340-7436

Conseiller technique, Services du savoir, de l'information et de la technologie
Conseil national de recherches Canada / Gouvernement du Canada
joey.dum...@nrc-cnrc.gc.ca / Tél.: 
613-990-8152 / Tél. cell.: 438-340-7436

From: pulp-list-boun...@redhat.com 
mailto:pulp-list-boun...@redhat.com>> on behalf 
of Dumont, Joey mailto:joey.dum...@nrc-cnrc.gc.ca>>
Sent: 06 November 2019 11:36
To: Mike DePaulo
Cc: pulp-list@redhat.com
Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on 
Amazon EC2 (pulp3)


PR, as suggested: https://github.com/pulp/pulpcore/pull/366.


Cheers,


Joey Dumont

Technical Advisor, Knowledge, Information, and Technology Services
National Research Council Canada / Governement of Canada
joey.dum...@nrc-cnrc.gc.ca / Tel: 
613-990-8152 / Cell: 438-340-7436

Conseiller technique, Services du savoir, de l'information et de la technologie
Conseil national de recherches Canada / Gouvernement du Canada
joey.dum...@nrc-cnrc.gc.ca / Tél.: 
613-990-8152 / Tél. cell.: 438-340-7436

From: Mike DePaulo mailto:mikedep...@redhat.com>>
Sent: 06 November 2019 10:52
To: Dumont, Joey
Cc: David Davis; pulp-list@redhat.com
Subject: Re: [Pulp-list] Issues with using S3 storage when running pulp on 
Amazon EC2 (pulp3)

Hi Joey,

It sounds like aws_default_acl should be documented here then:
https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws

Care to submit a documentation PR?
https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst

Thanks,
-Mike

On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey 
mailto:joey.dum...@nrc-cnrc.gc.ca>> wrote:
Turns out the issue was on my end. I had to add aws_default_acl: None to the 
pulp_settings section of the playbook. The public-read ACL was incompatible 
with the BlockPublicAccess settings that I had set on my S3 bucket. 

I'm now encountering a different issue, but I'll start another thread for that 
one.

Thanks for the pointers, they were very helpful!

Joey Dumont

Technical Advisor, Knowledge, Information, and Technology Services
National Research Council Canada / Governement of Canada
joey.dum...@nrc-cnrc.gc.ca / Tel: 
613-990-8152 / Cell: 438-340-7436

Conseiller technique, Services du savoir, de l'information et de la technologie
Conseil national de

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

2019-11-09 Thread Mike DePaulo

Hmm, I'm not sure if this should be addressed via dynaconf somehow or via
Ansible logic.

For troubleshhooting purposes, we can bypass the Ansible part by modify the
settings.py manually, after it gets written (for example, with a
non-standard Ansible task you add.)

-Mike

On Sat, Nov 9, 2019, 2:05 PM Dumont, Joey 
wrote:

> I have tried '~', 'null' and '' (without the quotes), and they all get
> rendered as null. This leads to an issue with running the migrations when
> running the Ansible script.
>
>
> I opened an issue: https://pulp.plan.io/issues/5687
>
>
> Maybe this should have gone in the pulp-ansible project?
>
>
> Cheers,
>
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> --
> *From:* Mike DePaulo 
> *Sent:* 09 November 2019 13:47
> *To:* David Davis
> *Cc:* Dumont, Joey; pulp-list@redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
> I think that with Ansible syntax, you need to specify:
>   AWS_DEFAULT_ACL:
>
> I am fairly certain the lack of any value at the end would get interpreted
> as Python None.
>
> - Mike
>
> On Sat, Nov 9, 2019 at 6:57 AM David Davis  wrote:
>
>> You should be able to set aws_default_acl to null (ie "aws_default_acl:
>> ~" or "aws_default_acl: null"). Sounds like that's not working though. What
>> error did you get? Would you mind filing an issue?
>>
>> https://pulp.plan.io/projects/pulp/issues/new
>>
>> Thanks.
>>
>> David
>>
>>
>> On Fri, Nov 8, 2019 at 10:20 PM Dumont, Joey 
>> wrote:
>>
>>> I encounted a new issue that escaped my notice by some weird coincidence.
>>>
>>>
>>> It turns that you use the pulp_settings section of the playbook,
>>> aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead
>>> of AWS_DEFAULT_ACL = None, which results in the cryptic
>>>
>>>
>>> "An error occurred (InvalidArgument) when calling the PutObject
>>> operation: None"
>>>
>>> Is there a way to ensure that Ansible converts aws_default_acl: None to
>>> AWS_DEFAULT_ACL = None?
>>>
>>> I tried setting it to null but I got an error.
>>>
>>> Cheers,
>>> 
>>>
>>>
>>> Joey Dumont
>>>
>>> Technical Advisor, Knowledge, Information, and Technology Services
>>> National Research Council Canada / Governement of Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>
>>> Conseiller technique, Services du savoir, de l'information et de la
>>> technologie
>>> Conseil national de recherches Canada / Gouvernement du Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>> 438-340-7436
>>> --
>>> *From:* pulp-list-boun...@redhat.com  on
>>> behalf of Dumont, Joey 
>>> *Sent:* 06 November 2019 11:36
>>> *To:* Mike DePaulo
>>> *Cc:* pulp-list@redhat.com
>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>> pulp on Amazon EC2 (pulp3)
>>>
>>>
>>> PR, as suggested: https://github.com/pulp/pulpcore/pull/366.
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Joey Dumont
>>>
>>> Technical Advisor, Knowledge, Information, and Technology Services
>>> National Research Council Canada / Governement of Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>
>>> Conseiller technique, Services du savoir, de l'information et de la
>>> technologie
>>> Conseil national de recherches Canada / Gouvernement du Canada
>>> joey.dum...@nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>> 438-340-7436
>>> --
>>> *From:* Mike DePaulo 
>>> *Sent:* 06 November 2019 10:52
>>> *To:* Dumont, Joey
>>> *Cc:* David Davis; pulp-list@redhat.com
>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>> pulp on Amazon EC2 (pulp3)
>>>
>>> Hi Joey,
>>>
>>> It sounds like aws_default_acl should be documented here then:
>>>
>>> https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws
>>>
>>> Care to submit a documentation PR?
>>>
>>> https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst
>>>
>>> Thanks,
>>> -Mike
>>>
>>> On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey 
>>> wrote:
>>>
 Turns out the issue was on my end. I had to add aws_default_acl: None
 to the pulp_settings section of the playbook. The public-read ACL was
 incompatible with the BlockPublicAccess settings that I had set on my S3
 bucket. 

 I'm now encountering a different issue, but I'll start another thread
 for that one.

 Thanks for the pointers, they were very helpful!

 Joey Dumont

 Technical Advisor, Knowledge, Information,

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

Re: [Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

4 matches

Site Navigation

Mail list logo

Footer information