Re: [Pulp-dev] Importer Name

[David Davis]

I was one of the voices of dissent on this but I am more of a -0/+0 than a
-1.


David

On Fri, Mar 23, 2018 at 2:56 PM, Dana Walker  wrote:

> FWIW, I've looked them over and think they're good, but given the +/- 1's
> discussion on IRC, you may want to get further grooming consensus still.
>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> On Fri, Mar 23, 2018 at 1:51 PM, Austin Macdonald 
> wrote:
>
>> Would anyone mind grooming these? Also, last change to shut it down :)
>> https://pulp.plan.io/issues/3488
>> https://pulp.plan.io/issues/3492
>>
>>
>>
>> On Fri, Mar 16, 2018 at 9:27 AM, Austin Macdonald 
>> wrote:
>>
>>> Ive created tasks to do this work.
>>>
>>> Rename:
>>> https://pulp.plan.io/issues/3488
>>>
>>> Move sync_mode, remove download_policy
>>> https://pulp.plan.io/issues/3492
>>>
>>> Each of these has corresponding plugin tasks, which are related to them.
>>>
>>> On Thu, Mar 15, 2018 at 4:16 PM, David Davis 
>>> wrote:
>>>
 Austin and Jeff,

 Thanks for the responses. I am happy with moving forward and opening an
 issue in redmine for this change

 I think I am +0 on dropping the fields. However, if we start to get
 complaints from our users, I think we should consider adding them back.


 David

 On Wed, Mar 14, 2018 at 10:16 AM, Jeff Ortel  wrote:

> In pulp3, users need to keep track for a number of things.  For
> example, without auto publish, users need to keep track of which
> importer(s) and publishers need to be used for sync/publish workflows.  I
> fully expect that users using the API will be maintaining some kind of
> automation/orchestration on their end (shell scripts, ansible).  So,
> keeping track of sync and download policies does not seem like much of a
> burden.  Also, after further consideration, I don't think that storing
> either the sync (mode) or download policy on the repository is
> appropriate.
>
>
> On 03/13/2018 04:59 PM, David Davis wrote:
>
> Can you elaborate on what made you reconsider? Asking because I still
> see the point that you and Justin raised about dropping the fields as an
> issue.
>
>
> David
>
> On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel 
> wrote:
>
>>
>>
>> On 03/12/2018 10:28 AM, Jeff Ortel wrote:
>>
>> On 03/08/2018 10:13 AM, Austin Macdonald wrote:
>>
>> Motivation:
>> The name "importer" carries some inaccurate implications.
>> 1) Importers should "import". Tasks like "sync" will do the actual
>> importing. The object only holds the configuration that happens to be 
>> used
>> by sync tasks.
>> 2) Sync tasks on mirror mode remove content as well as add it, so
>> "import" isn't quite right.
>>
>> Proposed name: Remote
>>
>> The inspiration for remote is "git remote". In git, remotes represent
>> external repositories, which is almost exactly what our importers do.
>>
>>
>> +1, The git/ostree "remote" concept applies very well to most of what
>> an "importer" defines in pulp.
>>
>>
>> ---
>> Part 2: Trim the fields
>>
>> Currently, Importers have settings that can be categorized in 2 ways.
>> I am proposing removing the "sync settings" from the Remote model:
>>
>> External Source information
>> name
>> feed_url
>> validate
>> ssl_ca_certificate
>> ssl_client_certificate
>> ssl_client_key
>> ssl_validation
>> proxy_url
>> username
>> password
>>
>> Sync settings
>> download_policy
>> sync_mode
>>
>> This had some advantages when Importers were related to Repositories.
>> For example, having a repository.importer that always used the same sync
>> mode made sense. However, the "how" to sync settings don't make much 
>> sense
>> when importers and repositories are not linked. It seems very reasonable
>> that a user might have 2 repositories that sync from the same source (ex
>> EPEL). It does not make sense for them to have create an Importer for the
>> EPEL repository twice or more just to change sync_mode or download 
>> policy.
>> Instead of modeling these fields, I propose that they should POST body
>> parameters.
>>
>>
>> I, as a user, don't like having to specify download_policy &
>> sync_mode on every request.  The burden on the user to passing these
>> consistently seems unnecessary and prone to error.  And, like something
>> that pulp should store as part of it's value proposition.   Imagine an
>> organization with tons of repositories and 

Re: [Pulp-dev] Importer Name

[Dana Walker]

FWIW, I've looked them over and think they're good, but given the +/- 1's
discussion on IRC, you may want to get further grooming consensus still.

--Dana

Dana Walker

Associate Software Engineer

Red Hat




On Fri, Mar 23, 2018 at 1:51 PM, Austin Macdonald 
wrote:

> Would anyone mind grooming these? Also, last change to shut it down :)
> https://pulp.plan.io/issues/3488
> https://pulp.plan.io/issues/3492
>
>
>
> On Fri, Mar 16, 2018 at 9:27 AM, Austin Macdonald 
> wrote:
>
>> Ive created tasks to do this work.
>>
>> Rename:
>> https://pulp.plan.io/issues/3488
>>
>> Move sync_mode, remove download_policy
>> https://pulp.plan.io/issues/3492
>>
>> Each of these has corresponding plugin tasks, which are related to them.
>>
>> On Thu, Mar 15, 2018 at 4:16 PM, David Davis 
>> wrote:
>>
>>> Austin and Jeff,
>>>
>>> Thanks for the responses. I am happy with moving forward and opening an
>>> issue in redmine for this change
>>>
>>> I think I am +0 on dropping the fields. However, if we start to get
>>> complaints from our users, I think we should consider adding them back.
>>>
>>>
>>> David
>>>
>>> On Wed, Mar 14, 2018 at 10:16 AM, Jeff Ortel  wrote:
>>>
 In pulp3, users need to keep track for a number of things.  For
 example, without auto publish, users need to keep track of which
 importer(s) and publishers need to be used for sync/publish workflows.  I
 fully expect that users using the API will be maintaining some kind of
 automation/orchestration on their end (shell scripts, ansible).  So,
 keeping track of sync and download policies does not seem like much of a
 burden.  Also, after further consideration, I don't think that storing
 either the sync (mode) or download policy on the repository is
 appropriate.


 On 03/13/2018 04:59 PM, David Davis wrote:

 Can you elaborate on what made you reconsider? Asking because I still
 see the point that you and Justin raised about dropping the fields as an
 issue.


 David

 On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel  wrote:

>
>
> On 03/12/2018 10:28 AM, Jeff Ortel wrote:
>
> On 03/08/2018 10:13 AM, Austin Macdonald wrote:
>
> Motivation:
> The name "importer" carries some inaccurate implications.
> 1) Importers should "import". Tasks like "sync" will do the actual
> importing. The object only holds the configuration that happens to be used
> by sync tasks.
> 2) Sync tasks on mirror mode remove content as well as add it, so
> "import" isn't quite right.
>
> Proposed name: Remote
>
> The inspiration for remote is "git remote". In git, remotes represent
> external repositories, which is almost exactly what our importers do.
>
>
> +1, The git/ostree "remote" concept applies very well to most of what
> an "importer" defines in pulp.
>
>
> ---
> Part 2: Trim the fields
>
> Currently, Importers have settings that can be categorized in 2 ways.
> I am proposing removing the "sync settings" from the Remote model:
>
> External Source information
> name
> feed_url
> validate
> ssl_ca_certificate
> ssl_client_certificate
> ssl_client_key
> ssl_validation
> proxy_url
> username
> password
>
> Sync settings
> download_policy
> sync_mode
>
> This had some advantages when Importers were related to Repositories.
> For example, having a repository.importer that always used the same sync
> mode made sense. However, the "how" to sync settings don't make much sense
> when importers and repositories are not linked. It seems very reasonable
> that a user might have 2 repositories that sync from the same source (ex
> EPEL). It does not make sense for them to have create an Importer for the
> EPEL repository twice or more just to change sync_mode or download policy.
> Instead of modeling these fields, I propose that they should POST body
> parameters.
>
>
> I, as a user, don't like having to specify download_policy &
> sync_mode on every request.  The burden on the user to passing these
> consistently seems unnecessary and prone to error.  And, like something
> that pulp should store as part of it's value proposition.   Imagine an
> organization with tons of repositories and admins.  They would need to
> maintain a spreadsheet, notes, scripts for these settings so that admin A
> is syncing using the same settings as admin B.
>
> Perhaps download_policy &  sync_mode should be attributes of the
> repository.  Thoughts on moving them there.  The sync_mode
> (mirror/additive) may need to be 

Re: [Pulp-dev] Importer Name

[Austin Macdonald]

Ive created tasks to do this work.

Rename:
https://pulp.plan.io/issues/3488

Move sync_mode, remove download_policy
https://pulp.plan.io/issues/3492

Each of these has corresponding plugin tasks, which are related to them.

On Thu, Mar 15, 2018 at 4:16 PM, David Davis  wrote:

> Austin and Jeff,
>
> Thanks for the responses. I am happy with moving forward and opening an
> issue in redmine for this change
>
> I think I am +0 on dropping the fields. However, if we start to get
> complaints from our users, I think we should consider adding them back.
>
>
> David
>
> On Wed, Mar 14, 2018 at 10:16 AM, Jeff Ortel  wrote:
>
>> In pulp3, users need to keep track for a number of things.  For example,
>> without auto publish, users need to keep track of which importer(s) and
>> publishers need to be used for sync/publish workflows.  I fully expect that
>> users using the API will be maintaining some kind of automation/
>> orchestration on their end (shell scripts, ansible).  So, keeping track
>> of sync and download policies does not seem like much of a burden.  Also,
>> after further consideration, I don't think that storing either the sync
>> (mode) or download policy on the repository is appropriate.
>>
>>
>> On 03/13/2018 04:59 PM, David Davis wrote:
>>
>> Can you elaborate on what made you reconsider? Asking because I still see
>> the point that you and Justin raised about dropping the fields as an issue.
>>
>>
>> David
>>
>> On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 03/12/2018 10:28 AM, Jeff Ortel wrote:
>>>
>>> On 03/08/2018 10:13 AM, Austin Macdonald wrote:
>>>
>>> Motivation:
>>> The name "importer" carries some inaccurate implications.
>>> 1) Importers should "import". Tasks like "sync" will do the actual
>>> importing. The object only holds the configuration that happens to be used
>>> by sync tasks.
>>> 2) Sync tasks on mirror mode remove content as well as add it, so
>>> "import" isn't quite right.
>>>
>>> Proposed name: Remote
>>>
>>> The inspiration for remote is "git remote". In git, remotes represent
>>> external repositories, which is almost exactly what our importers do.
>>>
>>>
>>> +1, The git/ostree "remote" concept applies very well to most of what an
>>> "importer" defines in pulp.
>>>
>>>
>>> ---
>>> Part 2: Trim the fields
>>>
>>> Currently, Importers have settings that can be categorized in 2 ways. I
>>> am proposing removing the "sync settings" from the Remote model:
>>>
>>> External Source information
>>> name
>>> feed_url
>>> validate
>>> ssl_ca_certificate
>>> ssl_client_certificate
>>> ssl_client_key
>>> ssl_validation
>>> proxy_url
>>> username
>>> password
>>>
>>> Sync settings
>>> download_policy
>>> sync_mode
>>>
>>> This had some advantages when Importers were related to Repositories.
>>> For example, having a repository.importer that always used the same sync
>>> mode made sense. However, the "how" to sync settings don't make much sense
>>> when importers and repositories are not linked. It seems very reasonable
>>> that a user might have 2 repositories that sync from the same source (ex
>>> EPEL). It does not make sense for them to have create an Importer for the
>>> EPEL repository twice or more just to change sync_mode or download policy.
>>> Instead of modeling these fields, I propose that they should POST body
>>> parameters.
>>>
>>>
>>> I, as a user, don't like having to specify download_policy &  sync_mode
>>> on every request.  The burden on the user to passing these consistently
>>> seems unnecessary and prone to error.  And, like something that pulp should
>>> store as part of it's value proposition.   Imagine an organization with
>>> tons of repositories and admins.  They would need to maintain a
>>> spreadsheet, notes, scripts for these settings so that admin A is syncing
>>> using the same settings as admin B.
>>>
>>> Perhaps download_policy &  sync_mode should be attributes of the
>>> repository.  Thoughts on moving them there.  The sync_mode
>>> (mirror/additive) may need to be renamed in a way that changes it from
>>> describing how the importer is syning to something that defines the type of
>>> repository.  Like that the repository is intended to be a mirror or not.
>>> Perhaps just a "mirror" (bool) attribute.
>>>
>>>
>>> I have reconsidered this.  Disregard.
>>>
>>>
>>> example
>>>
>>> POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive,
>>> dl_policy=immediate
>>> POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror,
>>> dl_policy=deferred
>>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing 
>>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> 

Re: [Pulp-dev] Importer Name

[David Davis]

Austin and Jeff,

Thanks for the responses. I am happy with moving forward and opening an
issue in redmine for this change

I think I am +0 on dropping the fields. However, if we start to get
complaints from our users, I think we should consider adding them back.


David

On Wed, Mar 14, 2018 at 10:16 AM, Jeff Ortel  wrote:

> In pulp3, users need to keep track for a number of things.  For example,
> without auto publish, users need to keep track of which importer(s) and
> publishers need to be used for sync/publish workflows.  I fully expect that
> users using the API will be maintaining some kind of automation/
> orchestration on their end (shell scripts, ansible).  So, keeping track
> of sync and download policies does not seem like much of a burden.  Also,
> after further consideration, I don't think that storing either the sync
> (mode) or download policy on the repository is appropriate.
>
>
> On 03/13/2018 04:59 PM, David Davis wrote:
>
> Can you elaborate on what made you reconsider? Asking because I still see
> the point that you and Justin raised about dropping the fields as an issue.
>
>
> David
>
> On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 03/12/2018 10:28 AM, Jeff Ortel wrote:
>>
>> On 03/08/2018 10:13 AM, Austin Macdonald wrote:
>>
>> Motivation:
>> The name "importer" carries some inaccurate implications.
>> 1) Importers should "import". Tasks like "sync" will do the actual
>> importing. The object only holds the configuration that happens to be used
>> by sync tasks.
>> 2) Sync tasks on mirror mode remove content as well as add it, so
>> "import" isn't quite right.
>>
>> Proposed name: Remote
>>
>> The inspiration for remote is "git remote". In git, remotes represent
>> external repositories, which is almost exactly what our importers do.
>>
>>
>> +1, The git/ostree "remote" concept applies very well to most of what an
>> "importer" defines in pulp.
>>
>>
>> ---
>> Part 2: Trim the fields
>>
>> Currently, Importers have settings that can be categorized in 2 ways. I
>> am proposing removing the "sync settings" from the Remote model:
>>
>> External Source information
>> name
>> feed_url
>> validate
>> ssl_ca_certificate
>> ssl_client_certificate
>> ssl_client_key
>> ssl_validation
>> proxy_url
>> username
>> password
>>
>> Sync settings
>> download_policy
>> sync_mode
>>
>> This had some advantages when Importers were related to Repositories. For
>> example, having a repository.importer that always used the same sync mode
>> made sense. However, the "how" to sync settings don't make much sense when
>> importers and repositories are not linked. It seems very reasonable that a
>> user might have 2 repositories that sync from the same source (ex EPEL). It
>> does not make sense for them to have create an Importer for the EPEL
>> repository twice or more just to change sync_mode or download policy.
>> Instead of modeling these fields, I propose that they should POST body
>> parameters.
>>
>>
>> I, as a user, don't like having to specify download_policy &  sync_mode
>> on every request.  The burden on the user to passing these consistently
>> seems unnecessary and prone to error.  And, like something that pulp should
>> store as part of it's value proposition.   Imagine an organization with
>> tons of repositories and admins.  They would need to maintain a
>> spreadsheet, notes, scripts for these settings so that admin A is syncing
>> using the same settings as admin B.
>>
>> Perhaps download_policy &  sync_mode should be attributes of the
>> repository.  Thoughts on moving them there.  The sync_mode
>> (mirror/additive) may need to be renamed in a way that changes it from
>> describing how the importer is syning to something that defines the type of
>> repository.  Like that the repository is intended to be a mirror or not.
>> Perhaps just a "mirror" (bool) attribute.
>>
>>
>> I have reconsidered this.  Disregard.
>>
>>
>> example
>>
>> POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive,
>> dl_policy=immediate
>> POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror,
>> dl_policy=deferred
>>
>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Jeff Ortel]

In pulp3, users need to keep track for a number of things.  For example, 
without auto publish, users need to keep track of which importer(s) and 
publishers need to be used for sync/publish workflows.  I fully expect 
that users using the API will be maintaining some kind of 
automation/orchestration on their end (shell scripts, ansible). So, 
keeping track of sync and download policies does not seem like much of a 
burden.  Also, after further consideration, I don't think that storing 
either the sync (mode) or download policy on the repository is appropriate.



On 03/13/2018 04:59 PM, David Davis wrote:
Can you elaborate on what made you reconsider? Asking because I still 
see the point that you and Justin raised about dropping the fields as 
an issue.



David

On Mon, Mar 12, 2018 at 12:31 PM, Jeff Ortel > wrote:




On 03/12/2018 10:28 AM, Jeff Ortel wrote:

On 03/08/2018 10:13 AM, Austin Macdonald wrote:

Motivation:
The name "importer" carries some inaccurate implications.
1) Importers should "import". Tasks like "sync" will do the
actual importing. The object only holds the configuration that
happens to be used by sync tasks.
2) Sync tasks on mirror mode remove content as well as add it,
so "import" isn't quite right.

Proposed name: Remote

The inspiration for remote is "git remote". In git, remotes
represent external repositories, which is almost exactly what
our importers do.


+1, The git/ostree "remote" concept applies very well to most of
what an "importer" defines in pulp.



---
Part 2: Trim the fields

Currently, Importers have settings that can be categorized in 2
ways. I am proposing removing the "sync settings" from the
Remote model:

External Source information
    name
    feed_url
    validate
    ssl_ca_certificate
    ssl_client_certificate
    ssl_client_key
    ssl_validation
    proxy_url
    username
    password

Sync settings
    download_policy
    sync_mode

This had some advantages when Importers were related to
Repositories. For example, having a repository.importer that
always used the same sync mode made sense. However, the "how" to
sync settings don't make much sense when importers and
repositories are not linked. It seems very reasonable that a
user might have 2 repositories that sync from the same source
(ex EPEL). It does not make sense for them to have create an
Importer for the EPEL repository twice or more just to change
sync_mode or download policy. Instead of modeling these fields,
I propose that they should POST body parameters.


I, as a user, don't like having to specify download_policy & 
sync_mode on every request.  The burden on the user to passing
these consistently seems unnecessary and prone to error.  And,
like something that pulp should store as part of it's value
proposition.   Imagine an organization with tons of repositories
and admins.  They would need to maintain a spreadsheet, notes,
scripts for these settings so that admin A is syncing using the
same settings as admin B.

Perhaps download_policy &  sync_mode should be attributes of the
repository.  Thoughts on moving them there.  The sync_mode
(mirror/additive) may need to be renamed in a way that changes it
from describing how the importer is syning to something that
defines the type of repository.  Like that the repository is
intended to be a mirror or not. Perhaps just a "mirror" (bool)
attribute.


I have reconsidered this.  Disregard.



example
POST v3/remotes/1234/sync/ repositorty=myrepo_href
sync_mode=additive, dl_policy=immediate
POST v3/remotes/1234/sync/ repositorty=myother_href
sync_mode=mirror, dl_policy=deferred



___
Pulp-dev mailing list
Pulp-dev@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-dev






___
Pulp-dev mailing list
Pulp-dev@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-dev





___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Austin Macdonald]

>
> On Mon, Mar 12, 2018 at 9:50 AM, Justin Sherrill 
>  wrote:
>

>> I'm fairly apathetic to a name change.  It would be annoying to us in
>> katello land, but not really a huge deal either way.  I don't think
>> importer a bad name, as it does hold configuration around 'importing'.  the
>> fact that it itself doesn't actually do the importing is a technical detail
>> and really isn't a big deal IMO.   User's likely wouldn't care, but for
>> developers i guess its just weighing a more 'perfect' name to the work of
>> changing everything (including documentation) at this stage.
>>
>> The reason for making this change despite it being "one more thing" is
that the subtle changes we have already made changes the role of Importers
significantly. When Importers were one-to-one with Repositories, it was
reasonable to assume that sync_mode and download_policy would be the same
for each sync. However, without that relationship, multiple repositories
can "sync from" a single external source (hopefully modeled as a Remote)
and I don't think it makes sense to assume that they will use the same sync
settings.


> So as a user using some future cli, i have to magically 'remember' these
>> values?  If so, that seems like a bad user experience and kinda defeats the
>> purpose of pulp holding configuration.  Could it be stored on the
>> repository itself (or somewhere else) if it doesn't make sense to store on
>> the importer/remote?
>>
>> If you're serious about this, this would be something to ask current
>> users of pulp as it seems kind of a big deal.
>>
>
You are right about this problem, and while I want to change how this will
work at a base level, it should eventually be abstracted for the end-user
(more on this below). Unfortunately, the CLI usability problem will exist
even if we leave the sync settings on the Importer/Remote. Objects in Pulp
3 are referenced *exclusively* by href, which uses a UUID (not natural key)
and should not be generated by users. This is the main reason I have argued
that we should not release an auto-generated CLI, because it would be (IMO)
less useful than httpie. One long term partial solution to this is a CLI
that bundles multiple commands. For instance, the user would specify repo
and importer names and the CLI would retrieve the appropriate href from the
rest api and then make the call.

A simpler option (-0 from me) would be just to move sync_mode from the
Importer to the Repository, but I don't think this would play very well for
repos that sync with multiple importers.

This doesn't address the concern of having to remember sync_mode though.
For this issue, I have a couple of ideas. First is the possibility of
repeating a task (briefly mentioned in the "Plugin relationship to tasks"
thread). Another concept I've been considering is an option (again,
inspired by git) to "set-upstream" for a repository. This would provide
users a way to indicate that a particular repository is set up to sync from
a particular importer/remote using particular settings.this
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Jeff Ortel]

On 03/08/2018 10:13 AM, Austin Macdonald wrote:

Motivation:
The name "importer" carries some inaccurate implications.
1) Importers should "import". Tasks like "sync" will do the actual 
importing. The object only holds the configuration that happens to be 
used by sync tasks.
2) Sync tasks on mirror mode remove content as well as add it, so 
"import" isn't quite right.


Proposed name: Remote

The inspiration for remote is "git remote". In git, remotes represent 
external repositories, which is almost exactly what our importers do.


+1, The git/ostree "remote" concept applies very well to most of what an 
"importer" defines in pulp.




---
Part 2: Trim the fields

Currently, Importers have settings that can be categorized in 2 ways. 
I am proposing removing the "sync settings" from the Remote model:


External Source information
    name
    feed_url
    validate
    ssl_ca_certificate
    ssl_client_certificate
    ssl_client_key
    ssl_validation
    proxy_url
    username
    password

Sync settings
    download_policy
    sync_mode

This had some advantages when Importers were related to Repositories. 
For example, having a repository.importer that always used the same 
sync mode made sense. However, the "how" to sync settings don't make 
much sense when importers and repositories are not linked. It seems 
very reasonable that a user might have 2 repositories that sync from 
the same source (ex EPEL). It does not make sense for them to have 
create an Importer for the EPEL repository twice or more just to 
change sync_mode or download policy. Instead of modeling these fields, 
I propose that they should POST body parameters.


I, as a user, don't like having to specify download_policy & sync_mode 
on every request.  The burden on the user to passing these consistently 
seems unnecessary and prone to error.  And, like something that pulp 
should store as part of it's value proposition.   Imagine an 
organization with tons of repositories and admins. They would need to 
maintain a spreadsheet, notes, scripts for these settings so that admin 
A is syncing using the same settings as admin B.


Perhaps download_policy &  sync_mode should be attributes of the 
repository.  Thoughts on moving them there.  The sync_mode 
(mirror/additive) may need to be renamed in a way that changes it from 
describing how the importer is syning to something that defines the type 
of repository.  Like that the repository is intended to be a mirror or 
not.  Perhaps just a "mirror" (bool) attribute.


example
POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive, 
dl_policy=immediate
POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror, 
dl_policy=deferred




___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Justin Sherrill]

On 03/08/2018 11:13 AM, Austin Macdonald wrote:

Motivation:
The name "importer" carries some inaccurate implications.
1) Importers should "import". Tasks like "sync" will do the actual 
importing. The object only holds the configuration that happens to be 
used by sync tasks.
2) Sync tasks on mirror mode remove content as well as add it, so 
"import" isn't quite right.


Proposed name: Remote

The inspiration for remote is "git remote". In git, remotes represent 
external repositories, which is almost exactly what our importers do.
I'm fairly apathetic to a name change.  It would be annoying to us in 
katello land, but not really a huge deal either way.  I don't think 
importer a bad name, as it does hold configuration around 'importing'.  
the fact that it itself doesn't actually do the importing is a technical 
detail and really isn't a big deal IMO. User's likely wouldn't care, but 
for developers i guess its just weighing a more 'perfect' name to the 
work of changing everything (including documentation) at this stage.




---
Part 2: Trim the fields

Currently, Importers have settings that can be categorized in 2 ways. 
I am proposing removing the "sync settings" from the Remote model:


External Source information
    name
    feed_url
    validate
    ssl_ca_certificate
    ssl_client_certificate
    ssl_client_key
    ssl_validation
    proxy_url
    username
    password

Sync settings
    download_policy
    sync_mode

This had some advantages when Importers were related to Repositories. 
For example, having a repository.importer that always used the same 
sync mode made sense. However, the "how" to sync settings don't make 
much sense when importers and repositories are not linked. It seems 
very reasonable that a user might have 2 repositories that sync from 
the same source (ex EPEL). It does not make sense for them to have 
create an Importer for the EPEL repository twice or more just to 
change sync_mode or download policy. Instead of modeling these fields, 
I propose that they should POST body parameters.


example
POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive, 
dl_policy=immediate
POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror, 
dl_policy=deferred


So as a user using some future cli, i have to magically 'remember' these 
values?  If so, that seems like a bad user experience and kinda defeats 
the purpose of pulp holding configuration.  Could it be stored on the 
repository itself (or somewhere else) if it doesn't make sense to store 
on the importer/remote?


If you're serious about this, this would be something to ask current 
users of pulp as it seems kind of a big deal.




___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[David Davis]

I’m probably -0 on changing the name from Importer to Remote. I think
Remote is a better name but I worry about changing the data model too much
in respect to Pulp 2. The more we diverge from Pulp 2, the more work we
create for stuff like Katello. The word “importer" appears 500 times in
their codebase. Sure, it might just be a find-and-replace in most cases but
when you combine all the other model changes we’ve made (repository
versions, unlinking publishers/importers from repos, using hrefs instead of
ids, etc) it adds up. I think getting Katello onto Pulp 3’s REST API and
creating a tool to migrate data from Pulp 2 is going to be a bit of a
nightmare.


David

On Sat, Mar 10, 2018 at 11:12 AM, Daniel Alley  wrote:

> +1 +1
>
> On Fri, Mar 9, 2018 at 3:28 PM, Brian Bouterse 
> wrote:
>
>> I left some responses inline.
>>
>> On Thu, Mar 8, 2018 at 11:13 AM, Austin Macdonald 
>> wrote:
>>
>>> Motivation:
>>> The name "importer" carries some inaccurate implications.
>>> 1) Importers should "import". Tasks like "sync" will do the actual
>>> importing. The object only holds the configuration that happens to be used
>>> by sync tasks.
>>> 2) Sync tasks on mirror mode remove content as well as add it, so
>>> "import" isn't quite right.
>>>
>>> Proposed name: Remote
>>>
>>> The inspiration for remote is "git remote". In git, remotes represent
>>> external repositories, which is almost exactly what our importers do.
>>>
>>
>> I'm +0 thinking that either name is fine but that those unfamiliar with
>> git, but those familiar with git would benefit from the name change.
>>
>>
>>>
>>> ---
>>> Part 2: Trim the fields
>>>
>>> Currently, Importers have settings that can be categorized in 2 ways. I
>>> am proposing removing the "sync settings" from the Remote model:
>>>
>>> External Source information
>>> name
>>> feed_url
>>> validate
>>> ssl_ca_certificate
>>> ssl_client_certificate
>>> ssl_client_key
>>> ssl_validation
>>> proxy_url
>>> username
>>> password
>>>
>>> Sync settings
>>> download_policy
>>> sync_mode
>>>
>>> This had some advantages when Importers were related to Repositories.
>>> For example, having a repository.importer that always used the same sync
>>> mode made sense. However, the "how" to sync settings don't make much sense
>>> when importers and repositories are not linked. It seems very reasonable
>>> that a user might have 2 repositories that sync from the same source (ex
>>> EPEL). It does not make sense for them to have create an Importer for the
>>> EPEL repository twice or more just to change sync_mode or download policy.
>>> Instead of modeling these fields, I propose that they should POST body
>>> parameters.
>>>
>>> example
>>>
>>> POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive,
>>> dl_policy=immediate
>>> POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror,
>>> dl_policy=deferred
>>>
>>
>> +1 to this change. I think this makes sense to specify the sync_mode and
>> dl_policy with sync params with each download because the Importer having
>> one policy apply to any repo that uses it does not make much sense.
>>
>>
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Daniel Alley]

+1 +1

On Fri, Mar 9, 2018 at 3:28 PM, Brian Bouterse  wrote:

> I left some responses inline.
>
> On Thu, Mar 8, 2018 at 11:13 AM, Austin Macdonald 
> wrote:
>
>> Motivation:
>> The name "importer" carries some inaccurate implications.
>> 1) Importers should "import". Tasks like "sync" will do the actual
>> importing. The object only holds the configuration that happens to be used
>> by sync tasks.
>> 2) Sync tasks on mirror mode remove content as well as add it, so
>> "import" isn't quite right.
>>
>> Proposed name: Remote
>>
>> The inspiration for remote is "git remote". In git, remotes represent
>> external repositories, which is almost exactly what our importers do.
>>
>
> I'm +0 thinking that either name is fine but that those unfamiliar with
> git, but those familiar with git would benefit from the name change.
>
>
>>
>> ---
>> Part 2: Trim the fields
>>
>> Currently, Importers have settings that can be categorized in 2 ways. I
>> am proposing removing the "sync settings" from the Remote model:
>>
>> External Source information
>> name
>> feed_url
>> validate
>> ssl_ca_certificate
>> ssl_client_certificate
>> ssl_client_key
>> ssl_validation
>> proxy_url
>> username
>> password
>>
>> Sync settings
>> download_policy
>> sync_mode
>>
>> This had some advantages when Importers were related to Repositories. For
>> example, having a repository.importer that always used the same sync mode
>> made sense. However, the "how" to sync settings don't make much sense when
>> importers and repositories are not linked. It seems very reasonable that a
>> user might have 2 repositories that sync from the same source (ex EPEL). It
>> does not make sense for them to have create an Importer for the EPEL
>> repository twice or more just to change sync_mode or download policy.
>> Instead of modeling these fields, I propose that they should POST body
>> parameters.
>>
>> example
>>
>> POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive,
>> dl_policy=immediate
>> POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror,
>> dl_policy=deferred
>>
>
> +1 to this change. I think this makes sense to specify the sync_mode and
> dl_policy with sync params with each download because the Importer having
> one policy apply to any repo that uses it does not make much sense.
>
>
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Importer Name

[Brian Bouterse]

I left some responses inline.

On Thu, Mar 8, 2018 at 11:13 AM, Austin Macdonald  wrote:

> Motivation:
> The name "importer" carries some inaccurate implications.
> 1) Importers should "import". Tasks like "sync" will do the actual
> importing. The object only holds the configuration that happens to be used
> by sync tasks.
> 2) Sync tasks on mirror mode remove content as well as add it, so "import"
> isn't quite right.
>
> Proposed name: Remote
>
> The inspiration for remote is "git remote". In git, remotes represent
> external repositories, which is almost exactly what our importers do.
>

I'm +0 thinking that either name is fine but that those unfamiliar with
git, but those familiar with git would benefit from the name change.


>
> ---
> Part 2: Trim the fields
>
> Currently, Importers have settings that can be categorized in 2 ways. I am
> proposing removing the "sync settings" from the Remote model:
>
> External Source information
> name
> feed_url
> validate
> ssl_ca_certificate
> ssl_client_certificate
> ssl_client_key
> ssl_validation
> proxy_url
> username
> password
>
> Sync settings
> download_policy
> sync_mode
>
> This had some advantages when Importers were related to Repositories. For
> example, having a repository.importer that always used the same sync mode
> made sense. However, the "how" to sync settings don't make much sense when
> importers and repositories are not linked. It seems very reasonable that a
> user might have 2 repositories that sync from the same source (ex EPEL). It
> does not make sense for them to have create an Importer for the EPEL
> repository twice or more just to change sync_mode or download policy.
> Instead of modeling these fields, I propose that they should POST body
> parameters.
>
> example
>
> POST v3/remotes/1234/sync/ repositorty=myrepo_href sync_mode=additive,
> dl_policy=immediate
> POST v3/remotes/1234/sync/ repositorty=myother_href sync_mode=mirror,
> dl_policy=deferred
>

+1 to this change. I think this makes sense to specify the sync_mode and
dl_policy with sync params with each download because the Importer having
one policy apply to any repo that uses it does not make much sense.


>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev