Re: [Pulp-dev] Pulp 3 REST API Challenges

[Milan Kovacik]

I believe option 1 is a better fit as I think a new repository version
is a side-effect of a (plug-in specific) action on a repository.

Cheers,
milan

On Mon, Apr 16, 2018 at 10:31 PM, Dennis Kliban  wrote:
> On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel  wrote:
>>
>>
>>
>> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>>
>> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>>>
>>>
>>>
>>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:



 On 04/10/2018 04:15 PM, Dennis Kliban wrote:

 On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
 wrote:
>
> These are good problem statements. I didn't understand all of the
> aspects of it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer
> to this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.
>

 I don't think all 3 are related problems. The motivation for grouping
 all together is that a subset of the action endpoints from problem 1 are
 used to create repository versions and Problem 3 is a problem with the
 repository version creation API.

>
>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
> wrote:
>>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>> Action endpoints are problematic.
>>
>> Example POST@/importers//sync/
>> They are non-RESTful and would make client code tightly coupled with
>> the server code.
>> These endpoints are inconsistent with the other parts of the REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe 
> "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not 
> RESTful"
> perhaps and have the self-consistency bullet removed?


 +1 to "Action endpoints are not RESTful"
 +1 to removing the self-consistency language

>>
>> DRF is not being used as intended for action endpoints so we have to
>> implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?
>
>>
>> We don't have a convention for where plug-in-specific, custom
>> repository version creation endpoints.
>>
>> example POST@/api/v3//docker/add/
>> needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed in
> the schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the 
> urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?
>


 Yes. I envision a CLI that can discover new plugin
 repository-version-creating functionality without having to install new
 client packages. Allowing plugin writers to add endpoints in arbitrary
 places for creating repository versions will make it impossible for the
 client to know what all the possible ways of creating a repository version
 are.

>>
>> For direct repository version creation, plugins are not involved.
>>
>> validation correctness problem: https://pulp.plan.io/issues/3541
>> example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
> plugin writers but not all.


 I think it affects all plugin writers. Even the File plugin needs to
 provide some validation when creating a repository version. Right now you
 can add a FileContent with the same relative path as another FileContent in
 the repository version. This should not be possible because it's not a 
 valid
 combination of FileContent units in the same repository version.


 Not necessarily.  Two files with the same relative path will have
 different digest (different content).  The assumption that they both cannot
 be in the same repository makes assumptions about how the repository is 
 used
 which I don't think is a good idea.  Image two different versions of
 abc.iso.

>>>
>>> Why is it bad to assume that a repository version is going

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

>
> *1.  Core endpoints do not delegate/redirect to plugins.*
>  - POST to /RepositoryVersion/ is removed.
>  - POST to /Publications/  (stays gone)
>  - Plugins provide endpoints for sync and other to create new
> repository versions.
>  - Plugins provide endpoints for creating Publications (publishing).
>

I think this is a good characterization of what would change. I'd like to
add my vision for why option 1 is optimal, leaving out my specific proposal.


   1. *Plugins manage their own data.* All actions
   (sync/publish/add/remove/etc) are controlled by the plugin. Core provides
   tools to make this trivial in the trivial cases, but when a plugin needs to
   do something differently, core stays out of the way. This is the only
   pattern that makes sense when dealing with every possible plugin. We can't
   afford to make assumptions about what every plugin will and won't need.
   2. *Plugin writers follow a single pattern. *Create a model, a viewset,
   and a serializer. If the default behavior doesn't work for you, override
   it. The parts are modular, so you probably only need small changes.
   3. *Every action has a separate endpoint, and complete documentation. *If
   file/sync, file/add/, python/add, are all separate endpoints, they all have
   separate documentation. Each endpoint's also documents its parameters.


> *2. Core delegates behavior to plugins for those endpoints requiring
> plugin participation.*
> - POST to /RepositoryVersion/ is the only way to create a repository
> version.
> - POST to /Publications/  is the only way to create a Publication
> (publish).
> - The POST parameters or body includes enough information so that core
> can select a plugin.
> - Either the entire POST is passed along to the plugin, *or* the
> plugin implements an API that's used by
>   core for pre-defined participation.
>

Restricting all repository version creation to a single endpoint is cool,
but Option 2 means that pulpcore accepts requests that involve plugins.
This begs the question, how does core involve plugins? This is such an
important question that it is difficult to discuss this idea at all outside
of a specific implementation.

There is one thing that really bothers me about any possible implementation
of Option 2, the documentation. If we combine various behaviors into a
single endpoint, then how do we clearly document each of them?
The plan for Option 2 as I understand it would not document (in the docs)
the specifics for how to sync/publish etc. Even if that problem is
addressed somehow, we still have an awkward hierarchy. As you can see
below, because plugins are primary in option 1, the action docs are
separated by plugin. This makes sense, because each workflow involves only
one plugin.

   - Option 1
  - As a plugin x user
 - I can sync.
 - I can add/remove
  - As a plugin y user
 - I can sync.
 - I can add/remove.
  - As a plugin z user
 - I can sync
 - I can add-complex-depsolving


   - Option 2
  -  As a user I can create a new RepositoryVersion
 - by plugin x sync
 - by plugin x add/remove
 - by plugin y sync
 - by plugin y add/remove
 - by plugin z sync
 - by plugin z add-complex-depsolving


There have been proposals on how both #1 and #2 can be achieved. I'm
>
wondering if we can even agree on one of these two approaches.
>

It would be very convenient if we agreed on an approach separately from
implementation, but I don't think it is possible. Each strategy has obvious
pros but the cons are tucked away in the implementation details.


>
>>
>>> A File repository version cannot be properly published if it contains 2
 FileContent units that both have the same relative path. The publisher has
 to decide which FileContent to publish at the relative path. That decision
 cannot be made intelligently by the publisher. The decision on which
 content unit to include in the repository version rests with the user that
 is creating the repository version. When a user tries to create a
 repository version with a FileContent that has the same relative path as
 another FileContent that was added previously, Pulp needs to inform the
 user that there is a conflict (and not create the repositiory version).
 This validation can only be provided by the File plugin.


>
>
>>
>>
>>> We would like to get feedback on these issues being sound and worth
>>> resolving before we resume particular solution discussion[1].
>>>
>>> Thanks,
>>> Austin, Dennis, and Milan
>>>
>>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066
>>> .html
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> _

Re: [Pulp-dev] Pulp 3 REST API Challenges

[David Davis]

+1 to option 2.

David

On Tue, Apr 17, 2018 at 9:39 AM, Brian Bouterse  wrote:

> I believe option 2 would be a good improvement for Pulp's API. It would
> resolve the current problem that a RepoVersion and/or a Publication can be
> created in multiple places in the REST API.
>
> On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>>
>> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:
>>>


 On 04/10/2018 04:15 PM, Dennis Kliban wrote:

 On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
 wrote:

> These are good problem statements. I didn't understand all of the
> aspects of it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer
> to this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.
>
>
 I don't think all 3 are related problems. The motivation for grouping
 all together is that a subset of the action endpoints from problem 1 are
 used to create repository versions and Problem 3 is a problem with the
 repository version creation API.


>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
> wrote:
>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>>- Action endpoints are problematic.
>>- Example POST@/importers//sync/
>>   - They are non-RESTful and would make client code tightly
>>   coupled with the server code.
>>   - These endpoints are inconsistent with the other parts of the
>>   REST API.
>>
>> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe 
> "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not 
> RESTful"
> perhaps and have the self-consistency bullet removed?
>

 +1 to "Action endpoints are not RESTful"
 +1 to removing the self-consistency language


>
>>- DRF is not being used as intended for action endpoints so we
>>   have to implement extra code. (against the grain)
>>
>> I don't know much about this. Where is the extra code?
>
>
>>
>>- We don't have a convention for where plug-in-specific, custom
>>repository version creation endpoints.
>>- example POST@/api/v3//docker/add/
>>   - needs to be discoverable through the schema
>>
>> What does discoverable via the schema ^ mean? Aren't all urls listed
> in the schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the 
> urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?
>
>

 Yes. I envision a CLI that can discover new plugin
 repository-version-creating functionality without having to install new
 client packages. Allowing plugin writers to add endpoints in arbitrary
 places for creating repository versions will make it impossible for the
 client to know what all the possible ways of creating a repository version
 are.


>
>>- For direct repository version creation, plugins are not
>>involved.
>>- validation correctness problem: https://pulp.plan.io/issues/3541
>>   - example: POST@/api/v3/repositories//versions/
>>
>> I agree with this problem statement. In terms of scope it affects
> some plugin writers but not all.
>

 I think it affects all plugin writers. Even the File plugin needs to
 provide some validation when creating a repository version. Right now you
 can add a FileContent with the same relative path as another FileContent in
 the repository version. This should not be possible because it's not a
 valid combination of FileContent units in the same repository version.


 Not necessarily.  Two files with the same relative path will have
 different digest (different content).  The assumption that they both cannot
 be in the same repository makes assumptions about how the repository is

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Brian Bouterse]

I believe option 2 would be a good improvement for Pulp's API. It would
resolve the current problem that a RepoVersion and/or a Publication can be
created in multiple places in the REST API.

On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel  wrote:

>
>
> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>
> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>
>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
>>> wrote:
>>>
 These are good problem statements. I didn't understand all of the
 aspects of it, so I put some inline questions.

 My overall question is: are these related problems? To share my answer
 to this, I believe the first two problems are related and the third is
 separate. The classic divide and conquor approach we could use here is to
 confirm that the problems are unrelated and focus on resolving one of them
 first.


>>> I don't think all 3 are related problems. The motivation for grouping
>>> all together is that a subset of the action endpoints from problem 1 are
>>> used to create repository versions and Problem 3 is a problem with the
>>> repository version creation API.
>>>
>>>

 On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
 wrote:

> Folks,
>
> Austin, Dennis, and Milan have identified the following issues with
> current Pulp3 REST API design:
>
>- Action endpoints are problematic.
>- Example POST@/importers//sync/
>   - They are non-RESTful and would make client code tightly
>   coupled with the server code.
>   - These endpoints are inconsistent with the other parts of the
>   REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
 consistency for REST since the "rest" of the API is RESTful. After reading
 parts of Roy Fielding's writeup of the definition of REST I believe "action
 endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
 are problematic" should be replaced with "Action endpoints are not RESTful"
 perhaps and have the self-consistency bullet removed?

>>>
>>> +1 to "Action endpoints are not RESTful"
>>> +1 to removing the self-consistency language
>>>
>>>

>- DRF is not being used as intended for action endpoints so we
>   have to implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?


>
>- We don't have a convention for where plug-in-specific, custom
>repository version creation endpoints.
>- example POST@/api/v3//docker/add/
>   - needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed
 in the schema?

 I think of ^ problem somewhat differently. Yes all urls need to be
 discoverable (a REST property), but isn't it more of an issue that the urls
 which produce repo versions can't be identified distinctly from any other
 plugin-contributed url? To paraphrase this perspective: making a repo
 version is strewn about throughout the API in random places which is a bad
 user experience. Is that what is motivation url discovery?


>>>
>>> Yes. I envision a CLI that can discover new plugin
>>> repository-version-creating functionality without having to install new
>>> client packages. Allowing plugin writers to add endpoints in arbitrary
>>> places for creating repository versions will make it impossible for the
>>> client to know what all the possible ways of creating a repository version
>>> are.
>>>
>>>

>- For direct repository version creation, plugins are not involved.
>- validation correctness problem: https://pulp.plan.io/issues/3541
>   - example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
 plugin writers but not all.

>>>
>>> I think it affects all plugin writers. Even the File plugin needs to
>>> provide some validation when creating a repository version. Right now you
>>> can add a FileContent with the same relative path as another FileContent in
>>> the repository version. This should not be possible because it's not a
>>> valid combination of FileContent units in the same repository version.
>>>
>>>
>>> Not necessarily.  Two files with the same relative path will have
>>> different digest (different content).  The assumption that they both cannot
>>> be in the same repository makes assumptions about how the repository is
>>> used which I don't think is a good idea.  Image two different versions of
>>> abc.iso.
>>>
>>>
>> Why is it bad to assume that a repository version is going to be
>> published? What are the other ways to use a repository version?
>>

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dennis Kliban]

On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel  wrote:

>
>
> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>
> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>
>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
>>> wrote:
>>>
 These are good problem statements. I didn't understand all of the
 aspects of it, so I put some inline questions.

 My overall question is: are these related problems? To share my answer
 to this, I believe the first two problems are related and the third is
 separate. The classic divide and conquor approach we could use here is to
 confirm that the problems are unrelated and focus on resolving one of them
 first.


>>> I don't think all 3 are related problems. The motivation for grouping
>>> all together is that a subset of the action endpoints from problem 1 are
>>> used to create repository versions and Problem 3 is a problem with the
>>> repository version creation API.
>>>
>>>

 On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
 wrote:

> Folks,
>
> Austin, Dennis, and Milan have identified the following issues with
> current Pulp3 REST API design:
>
>- Action endpoints are problematic.
>- Example POST@/importers//sync/
>   - They are non-RESTful and would make client code tightly
>   coupled with the server code.
>   - These endpoints are inconsistent with the other parts of the
>   REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
 consistency for REST since the "rest" of the API is RESTful. After reading
 parts of Roy Fielding's writeup of the definition of REST I believe "action
 endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
 are problematic" should be replaced with "Action endpoints are not RESTful"
 perhaps and have the self-consistency bullet removed?

>>>
>>> +1 to "Action endpoints are not RESTful"
>>> +1 to removing the self-consistency language
>>>
>>>

>- DRF is not being used as intended for action endpoints so we
>   have to implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?


>
>- We don't have a convention for where plug-in-specific, custom
>repository version creation endpoints.
>- example POST@/api/v3//docker/add/
>   - needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed
 in the schema?

 I think of ^ problem somewhat differently. Yes all urls need to be
 discoverable (a REST property), but isn't it more of an issue that the urls
 which produce repo versions can't be identified distinctly from any other
 plugin-contributed url? To paraphrase this perspective: making a repo
 version is strewn about throughout the API in random places which is a bad
 user experience. Is that what is motivation url discovery?


>>>
>>> Yes. I envision a CLI that can discover new plugin
>>> repository-version-creating functionality without having to install new
>>> client packages. Allowing plugin writers to add endpoints in arbitrary
>>> places for creating repository versions will make it impossible for the
>>> client to know what all the possible ways of creating a repository version
>>> are.
>>>
>>>

>- For direct repository version creation, plugins are not involved.
>- validation correctness problem: https://pulp.plan.io/issues/3541
>   - example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
 plugin writers but not all.

>>>
>>> I think it affects all plugin writers. Even the File plugin needs to
>>> provide some validation when creating a repository version. Right now you
>>> can add a FileContent with the same relative path as another FileContent in
>>> the repository version. This should not be possible because it's not a
>>> valid combination of FileContent units in the same repository version.
>>>
>>>
>>> Not necessarily.  Two files with the same relative path will have
>>> different digest (different content).  The assumption that they both cannot
>>> be in the same repository makes assumptions about how the repository is
>>> used which I don't think is a good idea.  Image two different versions of
>>> abc.iso.
>>>
>>>
>> Why is it bad to assume that a repository version is going to be
>> published? What are the other ways to use a repository version?
>>
>>
>> The repository may not be publish and/or may not be published by the
>> FilePublisher in the file plugin project.  A user may want to sync and
>> store many version of an iso in the repo

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/12/2018 04:49 PM, Dennis Kliban wrote:
On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel > wrote:




On 04/11/2018 01:13 PM, Dennis Kliban wrote:

On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel mailto:jor...@redhat.com>> wrote:



On 04/10/2018 04:15 PM, Dennis Kliban wrote:

On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse
mailto:bbout...@redhat.com>> wrote:

These are good problem statements. I didn't understand
all of the aspects of it, so I put some inline questions.

My overall question is: are these related problems? To
share my answer to this, I believe the first two
problems are related and the third is separate. The
classic divide and conquor approach we could use here is
to confirm that the problems are unrelated and focus on
resolving one of them first.


I don't think all 3 are related problems. The motivation for
grouping all together is that a subset of the action
endpoints from problem 1 are used to create repository
versions and Problem 3 is a problem with the repository
version creation API.


On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald
mailto:aus...@redhat.com>> wrote:

Folks,

Austin, Dennis, and Milan have identified the
following issues with current Pulp3 REST API design:

  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client
code tightly coupled with the server code.
  o These endpoints are inconsistent with the
other parts of the REST API.

Is self-consistency really a goal? I think it's a
placeholder for consistency for REST since the "rest" of
the API is RESTful. After reading parts of Roy
Fielding's writeup of the definition of REST I believe
"action endpoints are not RESTful" to be a true
statement. Maybe "Action endpoints are problematic"
should be replaced with "Action endpoints are not
RESTful" perhaps and have the self-consistency bullet
removed?


+1 to "Action endpoints are not RESTful"
+1 to removing the self-consistency language

  o DRF is not being used as intended for action
endpoints so we have to implement extra
code. (against the grain)

I don't know much about this. Where is the extra code?

  * We don't have a convention for where
plug-in-specific, custom repository version
creation endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema

What does discoverable via the schema ^ mean? Aren't all
urls listed in the schema?

I think of ^ problem somewhat differently. Yes all urls
need to be discoverable (a REST property), but isn't it
more of an issue that the urls which produce repo
versions can't be identified distinctly from any other
plugin-contributed url? To paraphrase this perspective:
making a repo version is strewn about throughout the API
in random places which is a bad user experience. Is that
what is motivation url discovery?


Yes. I envision a CLI that can discover new plugin
repository-version-creating functionality without having to
install new client packages. Allowing plugin writers to add
endpoints in arbitrary places for creating repository
versions will make it impossible for the client to know what
all the possible ways of creating a repository version are.

  * For direct repository version creation, plugins
are not involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example:
POST@/api/v3/repositories//versions/

I agree with this problem statement. In terms of scope
it affects some plugin writers but not all.


I think it affects all plugin writers. Even the File plugin
needs to provide some validation when creating a repository
version. Right now you can add a FileContent with the same
relative path as another FileContent in the repository
version. This should not be possible because it's not a
valid combination of FileContent units in the same
repository version.


Not necessarily.  Two files w

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dennis Kliban]

On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:

>
>
> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>
> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
> wrote:
>
>> These are good problem statements. I didn't understand all of the aspects
>> of it, so I put some inline questions.
>>
>> My overall question is: are these related problems? To share my answer to
>> this, I believe the first two problems are related and the third is
>> separate. The classic divide and conquor approach we could use here is to
>> confirm that the problems are unrelated and focus on resolving one of them
>> first.
>>
>>
> I don't think all 3 are related problems. The motivation for grouping all
> together is that a subset of the action endpoints from problem 1 are used
> to create repository versions and Problem 3 is a problem with the
> repository version creation API.
>
>
>>
>> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
>> wrote:
>>
>>> Folks,
>>>
>>> Austin, Dennis, and Milan have identified the following issues with
>>> current Pulp3 REST API design:
>>>
>>>- Action endpoints are problematic.
>>>- Example POST@/importers//sync/
>>>   - They are non-RESTful and would make client code tightly coupled
>>>   with the server code.
>>>   - These endpoints are inconsistent with the other parts of the
>>>   REST API.
>>>
>>> Is self-consistency really a goal? I think it's a placeholder for
>> consistency for REST since the "rest" of the API is RESTful. After reading
>> parts of Roy Fielding's writeup of the definition of REST I believe "action
>> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
>> are problematic" should be replaced with "Action endpoints are not RESTful"
>> perhaps and have the self-consistency bullet removed?
>>
>
> +1 to "Action endpoints are not RESTful"
> +1 to removing the self-consistency language
>
>
>>
>>>- DRF is not being used as intended for action endpoints so we have
>>>   to implement extra code. (against the grain)
>>>
>>> I don't know much about this. Where is the extra code?
>>
>>
>>>
>>>- We don't have a convention for where plug-in-specific, custom
>>>repository version creation endpoints.
>>>- example POST@/api/v3//docker/add/
>>>   - needs to be discoverable through the schema
>>>
>>> What does discoverable via the schema ^ mean? Aren't all urls listed in
>> the schema?
>>
>> I think of ^ problem somewhat differently. Yes all urls need to be
>> discoverable (a REST property), but isn't it more of an issue that the urls
>> which produce repo versions can't be identified distinctly from any other
>> plugin-contributed url? To paraphrase this perspective: making a repo
>> version is strewn about throughout the API in random places which is a bad
>> user experience. Is that what is motivation url discovery?
>>
>>
>
> Yes. I envision a CLI that can discover new plugin
> repository-version-creating functionality without having to install new
> client packages. Allowing plugin writers to add endpoints in arbitrary
> places for creating repository versions will make it impossible for the
> client to know what all the possible ways of creating a repository version
> are.
>
>
>>
>>>- For direct repository version creation, plugins are not involved.
>>>- validation correctness problem: https://pulp.plan.io/issues/3541
>>>   - example: POST@/api/v3/repositories//versions/
>>>
>>> I agree with this problem statement. In terms of scope it affects some
>> plugin writers but not all.
>>
>
> I think it affects all plugin writers. Even the File plugin needs to
> provide some validation when creating a repository version. Right now you
> can add a FileContent with the same relative path as another FileContent in
> the repository version. This should not be possible because it's not a
> valid combination of FileContent units in the same repository version.
>
>
> Not necessarily.  Two files with the same relative path will have
> different digest (different content).  The assumption that they both cannot
> be in the same repository makes assumptions about how the repository is
> used which I don't think is a good idea.  Image two different versions of
> abc.iso.
>
>
Why is it bad to assume that a repository version is going to be published?
What are the other ways to use a repository version?

A File repository version cannot be properly published if it contains 2
FileContent units that both have the same relative path. The publisher has
to decide which FileContent to publish at the relative path. That decision
cannot be made intelligently by the publisher. The decision on which
content unit to include in the repository version rests with the user that
is creating the repository version. When a user tries to create a
repository version with a FileContent that has the same relative path as
another FileContent that was added previously, Pulp needs to inform the
user that there is a conflict (and not create the repositiory

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Robin Chan]

Austin reported to me that this msg didn' show up in his inbox. I didn't
get it either.
https://www.redhat.com/archives/pulp-dev/2018-April/msg00136.html

I've opened up a ticket, so until then, we may need to check the archives.
https://www.redhat.com/archives/pulp-dev/index.html

On Thu, Apr 12, 2018 at 5:49 PM, Dennis Kliban  wrote:

> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>
>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
>>> wrote:
>>>
 These are good problem statements. I didn't understand all of the
 aspects of it, so I put some inline questions.

 My overall question is: are these related problems? To share my answer
 to this, I believe the first two problems are related and the third is
 separate. The classic divide and conquor approach we could use here is to
 confirm that the problems are unrelated and focus on resolving one of them
 first.


>>> I don't think all 3 are related problems. The motivation for grouping
>>> all together is that a subset of the action endpoints from problem 1 are
>>> used to create repository versions and Problem 3 is a problem with the
>>> repository version creation API.
>>>
>>>

 On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
 wrote:

> Folks,
>
> Austin, Dennis, and Milan have identified the following issues with
> current Pulp3 REST API design:
>
>- Action endpoints are problematic.
>- Example POST@/importers//sync/
>   - They are non-RESTful and would make client code tightly
>   coupled with the server code.
>   - These endpoints are inconsistent with the other parts of the
>   REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
 consistency for REST since the "rest" of the API is RESTful. After reading
 parts of Roy Fielding's writeup of the definition of REST I believe "action
 endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
 are problematic" should be replaced with "Action endpoints are not RESTful"
 perhaps and have the self-consistency bullet removed?

>>>
>>> +1 to "Action endpoints are not RESTful"
>>> +1 to removing the self-consistency language
>>>
>>>

>- DRF is not being used as intended for action endpoints so we
>   have to implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?


>
>- We don't have a convention for where plug-in-specific, custom
>repository version creation endpoints.
>- example POST@/api/v3//docker/add/
>   - needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed
 in the schema?

 I think of ^ problem somewhat differently. Yes all urls need to be
 discoverable (a REST property), but isn't it more of an issue that the urls
 which produce repo versions can't be identified distinctly from any other
 plugin-contributed url? To paraphrase this perspective: making a repo
 version is strewn about throughout the API in random places which is a bad
 user experience. Is that what is motivation url discovery?


>>>
>>> Yes. I envision a CLI that can discover new plugin
>>> repository-version-creating functionality without having to install new
>>> client packages. Allowing plugin writers to add endpoints in arbitrary
>>> places for creating repository versions will make it impossible for the
>>> client to know what all the possible ways of creating a repository version
>>> are.
>>>
>>>

>- For direct repository version creation, plugins are not involved.
>- validation correctness problem: https://pulp.plan.io/issues/3541
>   - example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
 plugin writers but not all.

>>>
>>> I think it affects all plugin writers. Even the File plugin needs to
>>> provide some validation when creating a repository version. Right now you
>>> can add a FileContent with the same relative path as another FileContent in
>>> the repository version. This should not be possible because it's not a
>>> valid combination of FileContent units in the same repository version.
>>>
>>>
>>> Not necessarily.  Two files with the same relative path will have
>>> different digest (different content).  The assumption that they both cannot
>>> be in the same repository makes assumptions about how the repository is
>>> used which I don't think is a good idea.  Image two different versions of
>>> abc.iso.
>>>
>>>
>> Why is it bad to assume that a repository version is going to be
>> published? What are the ot

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dennis Kliban]

On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:

>
>
> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>
> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:
>
>>
>>
>> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>>
>> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
>> wrote:
>>
>>> These are good problem statements. I didn't understand all of the
>>> aspects of it, so I put some inline questions.
>>>
>>> My overall question is: are these related problems? To share my answer
>>> to this, I believe the first two problems are related and the third is
>>> separate. The classic divide and conquor approach we could use here is to
>>> confirm that the problems are unrelated and focus on resolving one of them
>>> first.
>>>
>>>
>> I don't think all 3 are related problems. The motivation for grouping all
>> together is that a subset of the action endpoints from problem 1 are used
>> to create repository versions and Problem 3 is a problem with the
>> repository version creation API.
>>
>>
>>>
>>> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
>>> wrote:
>>>
 Folks,

 Austin, Dennis, and Milan have identified the following issues with
 current Pulp3 REST API design:

- Action endpoints are problematic.
- Example POST@/importers//sync/
   - They are non-RESTful and would make client code tightly
   coupled with the server code.
   - These endpoints are inconsistent with the other parts of the
   REST API.

 Is self-consistency really a goal? I think it's a placeholder for
>>> consistency for REST since the "rest" of the API is RESTful. After reading
>>> parts of Roy Fielding's writeup of the definition of REST I believe "action
>>> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
>>> are problematic" should be replaced with "Action endpoints are not RESTful"
>>> perhaps and have the self-consistency bullet removed?
>>>
>>
>> +1 to "Action endpoints are not RESTful"
>> +1 to removing the self-consistency language
>>
>>
>>>
- DRF is not being used as intended for action endpoints so we have
   to implement extra code. (against the grain)

 I don't know much about this. Where is the extra code?
>>>
>>>

- We don't have a convention for where plug-in-specific, custom
repository version creation endpoints.
- example POST@/api/v3//docker/add/
   - needs to be discoverable through the schema

 What does discoverable via the schema ^ mean? Aren't all urls listed in
>>> the schema?
>>>
>>> I think of ^ problem somewhat differently. Yes all urls need to be
>>> discoverable (a REST property), but isn't it more of an issue that the urls
>>> which produce repo versions can't be identified distinctly from any other
>>> plugin-contributed url? To paraphrase this perspective: making a repo
>>> version is strewn about throughout the API in random places which is a bad
>>> user experience. Is that what is motivation url discovery?
>>>
>>>
>>
>> Yes. I envision a CLI that can discover new plugin
>> repository-version-creating functionality without having to install new
>> client packages. Allowing plugin writers to add endpoints in arbitrary
>> places for creating repository versions will make it impossible for the
>> client to know what all the possible ways of creating a repository version
>> are.
>>
>>
>>>
- For direct repository version creation, plugins are not involved.
- validation correctness problem: https://pulp.plan.io/issues/3541
   - example: POST@/api/v3/repositories//versions/

 I agree with this problem statement. In terms of scope it affects some
>>> plugin writers but not all.
>>>
>>
>> I think it affects all plugin writers. Even the File plugin needs to
>> provide some validation when creating a repository version. Right now you
>> can add a FileContent with the same relative path as another FileContent in
>> the repository version. This should not be possible because it's not a
>> valid combination of FileContent units in the same repository version.
>>
>>
>> Not necessarily.  Two files with the same relative path will have
>> different digest (different content).  The assumption that they both cannot
>> be in the same repository makes assumptions about how the repository is
>> used which I don't think is a good idea.  Image two different versions of
>> abc.iso.
>>
>>
> Why is it bad to assume that a repository version is going to be
> published? What are the other ways to use a repository version?
>
>
> The repository may not be publish and/or may not be published by the
> FilePublisher in the file plugin project.  A user may want to sync and
> store many version of an iso in the repository and then selectively *add*
> a specific version to another repository for promotion work flows.  Also,
> the user could use another (custom) publisher that deals differently with
> multiple files with the same path in the 

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/11/2018 01:13 PM, Dennis Kliban wrote:
On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel > wrote:




On 04/10/2018 04:15 PM, Dennis Kliban wrote:

On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse
mailto:bbout...@redhat.com>> wrote:

These are good problem statements. I didn't understand all of
the aspects of it, so I put some inline questions.

My overall question is: are these related problems? To share
my answer to this, I believe the first two problems are
related and the third is separate. The classic divide and
conquor approach we could use here is to confirm that the
problems are unrelated and focus on resolving one of them first.


I don't think all 3 are related problems. The motivation for
grouping all together is that a subset of the action endpoints
from problem 1 are used to create repository versions and Problem
3 is a problem with the repository version creation API.


On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald
mailto:aus...@redhat.com>> wrote:

Folks,

Austin, Dennis, and Milan have identified the following
issues with current Pulp3 REST API design:

  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client code
tightly coupled with the server code.
  o These endpoints are inconsistent with the other
parts of the REST API.

Is self-consistency really a goal? I think it's a placeholder
for consistency for REST since the "rest" of the API is
RESTful. After reading parts of Roy Fielding's writeup of the
definition of REST I believe "action endpoints are not
RESTful" to be a true statement. Maybe "Action endpoints are
problematic" should be replaced with "Action endpoints are
not RESTful" perhaps and have the self-consistency bullet
removed?


+1 to "Action endpoints are not RESTful"
+1 to removing the self-consistency language

  o DRF is not being used as intended for action
endpoints so we have to implement extra code.
(against the grain)

I don't know much about this. Where is the extra code?

  * We don't have a convention for where
plug-in-specific, custom repository version creation
endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema

What does discoverable via the schema ^ mean? Aren't all urls
listed in the schema?

I think of ^ problem somewhat differently. Yes all urls need
to be discoverable (a REST property), but isn't it more of an
issue that the urls which produce repo versions can't be
identified distinctly from any other plugin-contributed url?
To paraphrase this perspective: making a repo version is
strewn about throughout the API in random places which is a
bad user experience. Is that what is motivation url discovery?


Yes. I envision a CLI that can discover new plugin
repository-version-creating functionality without having to
install new client packages. Allowing plugin writers to add
endpoints in arbitrary places for creating repository versions
will make it impossible for the client to know what all the
possible ways of creating a repository version are.

  * For direct repository version creation, plugins are
not involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example:
POST@/api/v3/repositories//versions/

I agree with this problem statement. In terms of scope it
affects some plugin writers but not all.


I think it affects all plugin writers. Even the File plugin needs
to provide some validation when creating a repository version.
Right now you can add a FileContent with the same relative path
as another FileContent in the repository version. This should not
be possible because it's not a valid combination of FileContent
units in the same repository version.


Not necessarily.  Two files with the same relative path will have
different digest (different content).  The assumption that they
both cannot be in the same repository makes assumptions about how
the repository is used which I don't think is a good idea.  Image
two different versions of abc.iso.


Why is it bad to assume that a repository version is going to be 
published? What are the other ways to use a repository version?


The repository may not be publish and/or may not be published by the 

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

>
>
- We don't have a convention for where plug-in-specific, custom
repository version creation endpoints.
- example POST@/api/v3//docker/add/
   - needs to be discoverable through the schema

 What does discoverable via the schema ^ mean? Aren't all urls listed in
>>> the schema?
>>>
>>> I think of ^ problem somewhat differently. Yes all urls need to be
>>> discoverable (a REST property), but isn't it more of an issue that the urls
>>> which produce repo versions can't be identified distinctly from any other
>>> plugin-contributed url? To paraphrase this perspective: making a repo
>>> version is strewn about throughout the API in random places which is a bad
>>> user experience. Is that what is motivation url discovery?
>>>
>>>
>>
>> Yes. I envision a CLI that can discover new plugin
>> repository-version-creating functionality without having to install new
>> client packages. Allowing plugin writers to add endpoints in arbitrary
>> places for creating repository versions will make it impossible for the
>> client to know what all the possible ways of creating a repository version
>> are.
>>
>>
>> Currently plugins can provide one (or more) arbitrary endpoints to
>> perform sync which is one form of creating a repository version.  How is
>> the endpoint for creating a version by adding content any different?
>>
>>
> I don't understand the question.
>
> Right now plugin writers can add REST API endpoints that create repository
> versions anywhere they want. This means the client has to inspect all of
> the API schema to find all the different ways to create a repository
> version. I would like to make this discovery process simpler for the
> client. It would be best if the client could learn about all the ways to
> create a repository version by making a single request to the API.
>

"Custom repository version creation endpoints need to be discoverable
through the schema."

We have some different ideas for how to implement this goal, and I think
that is coloring our perception of the problem as well. The statement is
vague, but we chose it so we could show what we agree on despite having
subtly different views.

The weakest interpretation of the statement is that every API endpoint is
documented in the schema. I doubt that anyone would disagree with that. In
case it isn't clear already, I want to restate that the scope of this point
is limited to endpoints that create repository versions and publications.
(sync, add, publish, complex copy, etc). Within that scope, I look at
"discoverability" as related to the consistency issue.

If we have a logical, consistent API, then users (including client users)
should be able to guess what the endpoint (or command) should be for a
given use case. This also includes the need for docs readers to have a good
idea where to find their information. That would obviously be difficult to
do with these endpoints spread all around the API. Dennis takes this a step
farther, suggesting that *any* action that creates a repository version,
regardless of plugin should be on a single endpoint. I suggest that
grouping them by plugin is enough. Obviously, this gets into the
implementation details a bit, so I don't want to get too much in the weeds
until we are back to discussing concrete proposals.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dennis Kliban]

On Tue, Apr 10, 2018 at 6:31 PM, Jeff Ortel  wrote:

>
>
> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>
> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
> wrote:
>
>> These are good problem statements. I didn't understand all of the aspects
>> of it, so I put some inline questions.
>>
>> My overall question is: are these related problems? To share my answer to
>> this, I believe the first two problems are related and the third is
>> separate. The classic divide and conquor approach we could use here is to
>> confirm that the problems are unrelated and focus on resolving one of them
>> first.
>>
>>
> I don't think all 3 are related problems. The motivation for grouping all
> together is that a subset of the action endpoints from problem 1 are used
> to create repository versions and Problem 3 is a problem with the
> repository version creation API.
>
>
>>
>> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
>> wrote:
>>
>>> Folks,
>>>
>>> Austin, Dennis, and Milan have identified the following issues with
>>> current Pulp3 REST API design:
>>>
>>>- Action endpoints are problematic.
>>>- Example POST@/importers//sync/
>>>   - They are non-RESTful and would make client code tightly coupled
>>>   with the server code.
>>>   - These endpoints are inconsistent with the other parts of the
>>>   REST API.
>>>
>>> Is self-consistency really a goal? I think it's a placeholder for
>> consistency for REST since the "rest" of the API is RESTful. After reading
>> parts of Roy Fielding's writeup of the definition of REST I believe "action
>> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
>> are problematic" should be replaced with "Action endpoints are not RESTful"
>> perhaps and have the self-consistency bullet removed?
>>
>
> +1 to "Action endpoints are not RESTful"
> +1 to removing the self-consistency language
>
>
>>
>>>- DRF is not being used as intended for action endpoints so we have
>>>   to implement extra code. (against the grain)
>>>
>>> I don't know much about this. Where is the extra code?
>>
>>
>>>
>>>- We don't have a convention for where plug-in-specific, custom
>>>repository version creation endpoints.
>>>- example POST@/api/v3//docker/add/
>>>   - needs to be discoverable through the schema
>>>
>>> What does discoverable via the schema ^ mean? Aren't all urls listed in
>> the schema?
>>
>> I think of ^ problem somewhat differently. Yes all urls need to be
>> discoverable (a REST property), but isn't it more of an issue that the urls
>> which produce repo versions can't be identified distinctly from any other
>> plugin-contributed url? To paraphrase this perspective: making a repo
>> version is strewn about throughout the API in random places which is a bad
>> user experience. Is that what is motivation url discovery?
>>
>>
>
> Yes. I envision a CLI that can discover new plugin
> repository-version-creating functionality without having to install new
> client packages. Allowing plugin writers to add endpoints in arbitrary
> places for creating repository versions will make it impossible for the
> client to know what all the possible ways of creating a repository version
> are.
>
>
> Currently plugins can provide one (or more) arbitrary endpoints to perform
> sync which is one form of creating a repository version.  How is the
> endpoint for creating a version by adding content any different?
>
>
I don't understand the question.

Right now plugin writers can add REST API endpoints that create repository
versions anywhere they want. This means the client has to inspect all of
the API schema to find all the different ways to create a repository
version. I would like to make this discovery process simpler for the
client. It would be best if the client could learn about all the ways to
create a repository version by making a single request to the API.


>
>
>
>>
>>>- For direct repository version creation, plugins are not involved.
>>>- validation correctness problem: https://pulp.plan.io/issues/3541
>>>   - example: POST@/api/v3/repositories//versions/
>>>
>>> I agree with this problem statement. In terms of scope it affects some
>> plugin writers but not all.
>>
>
> I think it affects all plugin writers. Even the File plugin needs to
> provide some validation when creating a repository version. Right now you
> can add a FileContent with the same relative path as another FileContent in
> the repository version. This should not be possible because it's not a
> valid combination of FileContent units in the same repository version.
>
>
>>
>>
>>> We would like to get feedback on these issues being sound and worth
>>> resolving before we resume particular solution discussion[1].
>>>
>>> Thanks,
>>> Austin, Dennis, and Milan
>>>
>>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://ww

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Austin Macdonald]

>
> Regarding the first bullet point, I think it’s an admirable goal but I
> worry that aiming for REST purity can cause us to do some funny things like
> defining fake resources that are really just actions on existing
> resources. It’s a bit like trying to be purely OO in Python and having to
> wrap a function in a fake class. I think this is what I found unappealing
> about the plugin tasks proposal.


This comment makes it sound like the Tasks proposal is forcing REST on the
design, but IMO our design is a great fit for REST. The fact is that Tasks
are resources no matter what, nothing fake about them. Whether we treat
them like all other resources is an open question. IMO we should only make
the API more RESTful if that also makes the API better.


> You mention consistency of REST but I found the plugin tasks proposal to
> lack consistency in certain areas. For example, some deletes are handled on
> resources (distributions, artifacts, etc) while other deletions would live
> in the tasks namespace (repos, publishers, etc).


That is not how it is designed. All object CRUD would stay where it is. We
are trying to focus on the problems and goals for now, but when the time
comes, I'll be sure to highlight this part of the design.

My overall question is: are these related problems?


When our goal is consistency, problems of related endpoints become related.
Jeff's earlier comment applies: "Looks like half of the plugins will need
to participate in creating repository versions (outside of sync). The API
design should take a consistent approach to creating repository versions (
*add* *and* *sync*)."


> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not RESTful"
> perhaps and have the self-consistency bullet removed?


Milan's "In my mind being RESTful implies consistency" +1. I value
consistency, REST is just one way to get it.

I think we are close to being able to move forward. It seems like there is
broad agreement about the goals/problems, with minor semantic differences.

On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:

>
>
> On 04/10/2018 04:15 PM, Dennis Kliban wrote:
>
> On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
> wrote:
>
>> These are good problem statements. I didn't understand all of the aspects
>> of it, so I put some inline questions.
>>
>> My overall question is: are these related problems? To share my answer to
>> this, I believe the first two problems are related and the third is
>> separate. The classic divide and conquor approach we could use here is to
>> confirm that the problems are unrelated and focus on resolving one of them
>> first.
>>
>>
> I don't think all 3 are related problems. The motivation for grouping all
> together is that a subset of the action endpoints from problem 1 are used
> to create repository versions and Problem 3 is a problem with the
> repository version creation API.
>
>
>>
>> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
>> wrote:
>>
>>> Folks,
>>>
>>> Austin, Dennis, and Milan have identified the following issues with
>>> current Pulp3 REST API design:
>>>
>>>- Action endpoints are problematic.
>>>- Example POST@/importers//sync/
>>>   - They are non-RESTful and would make client code tightly coupled
>>>   with the server code.
>>>   - These endpoints are inconsistent with the other parts of the
>>>   REST API.
>>>
>>> Is self-consistency really a goal? I think it's a placeholder for
>> consistency for REST since the "rest" of the API is RESTful. After reading
>> parts of Roy Fielding's writeup of the definition of REST I believe "action
>> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
>> are problematic" should be replaced with "Action endpoints are not RESTful"
>> perhaps and have the self-consistency bullet removed?
>>
>
> +1 to "Action endpoints are not RESTful"
> +1 to removing the self-consistency language
>
>
>>
>>>- DRF is not being used as intended for action endpoints so we have
>>>   to implement extra code. (against the grain)
>>>
>>> I don't know much about this. Where is the extra code?
>>
>>
>>>
>>>- We don't have a convention for where plug-in-specific, custom
>>>repository version creation endpoints.
>>>- example POST@/api/v3//docker/add/
>>>   - needs to be discoverable through the schema
>>>
>>> What does discoverable via the schema ^ mean? Aren't all urls listed in
>> the schema?
>>
>> I think of ^ problem somewhat differently. Yes all urls need to be
>> discoverable (a REST property), but isn't it more of an issue that the urls
>> which produce repo versions can't be identified distinctly from any other
>> plugin-contributed url? To p

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/10/2018 04:15 PM, Dennis Kliban wrote:
On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse > wrote:


These are good problem statements. I didn't understand all of the
aspects of it, so I put some inline questions.

My overall question is: are these related problems? To share my
answer to this, I believe the first two problems are related and
the third is separate. The classic divide and conquor approach we
could use here is to confirm that the problems are unrelated and
focus on resolving one of them first.


I don't think all 3 are related problems. The motivation for grouping 
all together is that a subset of the action endpoints from problem 1 
are used to create repository versions and Problem 3 is a problem with 
the repository version creation API.



On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald
mailto:aus...@redhat.com>> wrote:

Folks,

Austin, Dennis, and Milan have identified the following issues
with current Pulp3 REST API design:

  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client code
tightly coupled with the server code.
  o These endpoints are inconsistent with the other parts
of the REST API.

Is self-consistency really a goal? I think it's a placeholder for
consistency for REST since the "rest" of the API is RESTful. After
reading parts of Roy Fielding's writeup of the definition of REST
I believe "action endpoints are not RESTful" to be a true
statement. Maybe "Action endpoints are problematic" should be
replaced with "Action endpoints are not RESTful" perhaps and have
the self-consistency bullet removed?


+1 to "Action endpoints are not RESTful"
+1 to removing the self-consistency language

  o DRF is not being used as intended for action endpoints
so we have to implement extra code. (against the grain)

I don't know much about this. Where is the extra code?

  * We don't have a convention for where plug-in-specific,
custom repository version creation endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema

What does discoverable via the schema ^ mean? Aren't all urls
listed in the schema?

I think of ^ problem somewhat differently. Yes all urls need to be
discoverable (a REST property), but isn't it more of an issue that
the urls which produce repo versions can't be identified
distinctly from any other plugin-contributed url? To paraphrase
this perspective: making a repo version is strewn about throughout
the API in random places which is a bad user experience. Is that
what is motivation url discovery?


Yes. I envision a CLI that can discover new plugin 
repository-version-creating functionality without having to install 
new client packages. Allowing plugin writers to add endpoints in 
arbitrary places for creating repository versions will make it 
impossible for the client to know what all the possible ways of 
creating a repository version are.


  * For direct repository version creation, plugins are not
involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example:
POST@/api/v3/repositories//versions/

I agree with this problem statement. In terms of scope it affects
some plugin writers but not all.


I think it affects all plugin writers. Even the File plugin needs to 
provide some validation when creating a repository version. Right now 
you can add a FileContent with the same relative path as another 
FileContent in the repository version. This should not be possible 
because it's not a valid combination of FileContent units in the same 
repository version.


Not necessarily.  Two files with the same relative path will have 
different digest (different content).  The assumption that they both 
cannot be in the same repository makes assumptions about how the 
repository is used which I don't think is a good idea.  Image two 
different versions of abc.iso.




We would like to get feedback on these issues being sound and
worth resolving before we resume particular solution
discussion[1].

Thanks,
Austin, Dennis, and Milan

[1]
https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html



___
Pulp-dev mailing list
Pulp-dev@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-dev




___

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/10/2018 01:04 PM, Brian Bouterse wrote:
These are good problem statements. I didn't understand all of the 
aspects of it, so I put some inline questions.


My overall question is: are these related problems? To share my answer 
to this, I believe the first two problems are related and the third is 
separate. The classic divide and conquor approach we could use here is 
to confirm that the problems are unrelated and focus on resolving one 
of them first.


Agreed.




On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald > wrote:


Folks,

Austin, Dennis, and Milan have identified the following issues
with current Pulp3 REST API design:

  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client code tightly
coupled with the server code.
  o These endpoints are inconsistent with the other parts of
the REST API.

Is self-consistency really a goal? I think it's a placeholder for 
consistency for REST since the "rest" of the API is RESTful. After 
reading parts of Roy Fielding's writeup of the definition of REST I 
believe "action endpoints are not RESTful" to be a true statement. 
Maybe "Action endpoints are problematic" should be replaced with 
"Action endpoints are not RESTful" perhaps and have the 
self-consistency bullet removed?


Consistency in an API is always a valuable goal.


  o DRF is not being used as intended for action endpoints so
we have to implement extra code. (against the grain)

I don't know much about this. Where is the extra code?

  * We don't have a convention for where plug-in-specific, custom
repository version creation endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema

What does discoverable via the schema ^ mean? Aren't all urls listed 
in the schema?


I think of ^ problem somewhat differently. Yes all urls need to be 
discoverable (a REST property), but isn't it more of an issue that the 
urls which produce repo versions can't be identified distinctly from 
any other plugin-contributed url? To paraphrase this perspective: 
making a repo version is strewn about throughout the API in random 
places which is a bad user experience. Is that what is motivation url 
discovery?


Let me suggest another wording to "discovery".  The entire API needs to 
be clearly & completely defined in the schema.



  * For direct repository version creation, plugins are not involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example: POST@/api/v3/repositories//versions/

I agree with this problem statement. In terms of scope it affects some 
plugin writers but not all.


We would like to get feedback on these issues being sound and
worth resolving before we resume particular solution discussion[1].

Thanks,
Austin, Dennis, and Milan

[1]
https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html



___
Pulp-dev mailing list
Pulp-dev@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-dev





___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/10/2018 04:15 PM, Dennis Kliban wrote:
On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse > wrote:


These are good problem statements. I didn't understand all of the
aspects of it, so I put some inline questions.

My overall question is: are these related problems? To share my
answer to this, I believe the first two problems are related and
the third is separate. The classic divide and conquor approach we
could use here is to confirm that the problems are unrelated and
focus on resolving one of them first.


I don't think all 3 are related problems. The motivation for grouping 
all together is that a subset of the action endpoints from problem 1 
are used to create repository versions and Problem 3 is a problem with 
the repository version creation API.



On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald
mailto:aus...@redhat.com>> wrote:

Folks,

Austin, Dennis, and Milan have identified the following issues
with current Pulp3 REST API design:

  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client code
tightly coupled with the server code.
  o These endpoints are inconsistent with the other parts
of the REST API.

Is self-consistency really a goal? I think it's a placeholder for
consistency for REST since the "rest" of the API is RESTful. After
reading parts of Roy Fielding's writeup of the definition of REST
I believe "action endpoints are not RESTful" to be a true
statement. Maybe "Action endpoints are problematic" should be
replaced with "Action endpoints are not RESTful" perhaps and have
the self-consistency bullet removed?


+1 to "Action endpoints are not RESTful"
+1 to removing the self-consistency language

  o DRF is not being used as intended for action endpoints
so we have to implement extra code. (against the grain)

I don't know much about this. Where is the extra code?

  * We don't have a convention for where plug-in-specific,
custom repository version creation endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema

What does discoverable via the schema ^ mean? Aren't all urls
listed in the schema?

I think of ^ problem somewhat differently. Yes all urls need to be
discoverable (a REST property), but isn't it more of an issue that
the urls which produce repo versions can't be identified
distinctly from any other plugin-contributed url? To paraphrase
this perspective: making a repo version is strewn about throughout
the API in random places which is a bad user experience. Is that
what is motivation url discovery?


Yes. I envision a CLI that can discover new plugin 
repository-version-creating functionality without having to install 
new client packages. Allowing plugin writers to add endpoints in 
arbitrary places for creating repository versions will make it 
impossible for the client to know what all the possible ways of 
creating a repository version are.


Currently plugins can provide one (or more) arbitrary endpoints to 
perform sync which is one form of creating a repository version. How is 
the endpoint for creating a version by adding content any different?




  * For direct repository version creation, plugins are not
involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example:
POST@/api/v3/repositories//versions/

I agree with this problem statement. In terms of scope it affects
some plugin writers but not all.


I think it affects all plugin writers. Even the File plugin needs to 
provide some validation when creating a repository version. Right now 
you can add a FileContent with the same relative path as another 
FileContent in the repository version. This should not be possible 
because it's not a valid combination of FileContent units in the same 
repository version.



We would like to get feedback on these issues being sound and
worth resolving before we resume particular solution
discussion[1].

Thanks,
Austin, Dennis, and Milan

[1]
https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html



___
Pulp-dev mailing list
Pulp-dev@redhat.com 
https://www.redhat.com/mailman/listinfo/pulp-dev




___
Pulp-dev mailing list
Pulp-dev@redhat.com 

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dennis Kliban]

On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse  wrote:

> These are good problem statements. I didn't understand all of the aspects
> of it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer to
> this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.
>
>
I don't think all 3 are related problems. The motivation for grouping all
together is that a subset of the action endpoints from problem 1 are used
to create repository versions and Problem 3 is a problem with the
repository version creation API.


>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
> wrote:
>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>>- Action endpoints are problematic.
>>- Example POST@/importers//sync/
>>   - They are non-RESTful and would make client code tightly coupled
>>   with the server code.
>>   - These endpoints are inconsistent with the other parts of the
>>   REST API.
>>
>> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not RESTful"
> perhaps and have the self-consistency bullet removed?
>

+1 to "Action endpoints are not RESTful"
+1 to removing the self-consistency language


>
>>- DRF is not being used as intended for action endpoints so we have
>>   to implement extra code. (against the grain)
>>
>> I don't know much about this. Where is the extra code?
>
>
>>
>>- We don't have a convention for where plug-in-specific, custom
>>repository version creation endpoints.
>>- example POST@/api/v3//docker/add/
>>   - needs to be discoverable through the schema
>>
>> What does discoverable via the schema ^ mean? Aren't all urls listed in
> the schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?
>
>

Yes. I envision a CLI that can discover new plugin
repository-version-creating functionality without having to install new
client packages. Allowing plugin writers to add endpoints in arbitrary
places for creating repository versions will make it impossible for the
client to know what all the possible ways of creating a repository version
are.


>
>>- For direct repository version creation, plugins are not involved.
>>- validation correctness problem: https://pulp.plan.io/issues/3541
>>   - example: POST@/api/v3/repositories//versions/
>>
>> I agree with this problem statement. In terms of scope it affects some
> plugin writers but not all.
>

I think it affects all plugin writers. Even the File plugin needs to
provide some validation when creating a repository version. Right now you
can add a FileContent with the same relative path as another FileContent in
the repository version. This should not be possible because it's not a
valid combination of FileContent units in the same repository version.


>
>
>> We would like to get feedback on these issues being sound and worth
>> resolving before we resume particular solution discussion[1].
>>
>> Thanks,
>> Austin, Dennis, and Milan
>>
>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[David Davis]

Regarding the first bullet point, I think it’s an admirable goal but I
worry that aiming for REST purity can cause us to do some funny things like
defining fake resources that are really just actions on existing
resources. It’s a bit like trying to be purely OO in Python and having to
wrap a function in a fake class. I think this is what I found unappealing
about the plugin tasks proposal.

You mention consistency of REST but I found the plugin tasks proposal to
lack consistency in certain areas. For example, some deletes are handled on
resources (distributions, artifacts, etc) while other deletions would live
in the tasks namespace (repos, publishers, etc). So I am not sure how
moving to a strictly RESTful API solves all our consistency problems.

In my experience of working on RESTful APIs, we’ve always had to resort to
using action verbs so I’d be interested in seeing some other applications
that are purely RESTful if anyone has some examples.


David

On Tue, Apr 10, 2018 at 4:08 PM, Milan Kovacik  wrote:

> Brian,
>
> On Tue, Apr 10, 2018 at 8:04 PM, Brian Bouterse 
> wrote:
> > These are good problem statements. I didn't understand all of the
> aspects of
> > it, so I put some inline questions.
> >
> > My overall question is: are these related problems? To share my answer to
> > this, I believe the first two problems are related and the third is
> > separate. The classic divide and conquor approach we could use here is to
> > confirm that the problems are unrelated and focus on resolving one of
> them
> > first.
>
> Addressing any of these issues separately influences the solution to
> the rest of those therefore I believe them not to be independent.
> For instance, keeping action endpoint importer/sync/ while
> keeping POST@/repository//version/ while addressing the plug-in
> involvement absence, results in a design where plug-in configuration
> objects need to exist to be passed thru as parameters to those
> endpoints and causing inconsistency with plug-in specific version
> creation endpoints.
> At the same time we are aware of a solution that addresses all three
> issues at once by a design enhancement.
> What would be the motivation to enumerate the issues separately?
>
> >
> >
> > On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
> wrote:
> >>
> >> Folks,
> >>
> >> Austin, Dennis, and Milan have identified the following issues with
> >> current Pulp3 REST API design:
> >>
> >> Action endpoints are problematic.
> >>
> >> Example POST@/importers//sync/
> >> They are non-RESTful and would make client code tightly coupled with the
> >> server code.
> >> These endpoints are inconsistent with the other parts of the REST API.
> >
> > Is self-consistency really a goal? I think it's a placeholder for
> > consistency for REST since the "rest" of the API is RESTful. After
> reading
> > parts of Roy Fielding's writeup of the definition of REST I believe
> "action
> > endpoints are not RESTful" to be a true statement. Maybe "Action
> endpoints
> > are problematic" should be replaced with "Action endpoints are not
> RESTful"
> > perhaps and have the self-consistency bullet removed?
>
> In my mind being RESTful implies consistency but I might be wrong.
>
> >>
> >> DRF is not being used as intended for action endpoints so we have to
> >> implement extra code. (against the grain)
> >
> > I don't know much about this. Where is the extra code?
>
> for instance, custom action endpoints need custom documentation
> workarounds; see https://github.com/pulp/pulp_file/pull/60
> TL;DR: working around POST@/type/ that doesn't create an instance of the
> type.
>
> >
> >>
> >> We don't have a convention for where plug-in-specific, custom repository
> >> version creation endpoints.
> >>
> >> example POST@/api/v3//docker/add/
> >> needs to be discoverable through the schema
> >
> > What does discoverable via the schema ^ mean? Aren't all urls listed in
> the
> > schema?
> >
> > I think of ^ problem somewhat differently. Yes all urls need to be
> > discoverable (a REST property), but isn't it more of an issue that the
> urls
> > which produce repo versions can't be identified distinctly from any other
> > plugin-contributed url? To paraphrase this perspective: making a repo
> > version is strewn about throughout the API in random places which is a
> bad
> > user experience. Is that what is motivation url discovery?
>
> The point here is to stress the absence of a good rule-of-a-thumb.
> Currently POST@importers///sync/ and
> POST@/repository//version/ create repository version.
> Not being careful about a good design (by e.g preserving these core
> endpoints), custom plug-in actions might end up being yet another
> place where repository versions would be created.
>
> >
> >>
> >> For direct repository version creation, plugins are not involved.
> >>
> >> validation correctness problem: https://pulp.plan.io/issues/3541
> >> example: POST@/api/v3/repositories//versions/
> >
> > I agree with this problem statement. In terms of s

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Milan Kovacik]

Brian,

On Tue, Apr 10, 2018 at 8:04 PM, Brian Bouterse  wrote:
> These are good problem statements. I didn't understand all of the aspects of
> it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer to
> this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.

Addressing any of these issues separately influences the solution to
the rest of those therefore I believe them not to be independent.
For instance, keeping action endpoint importer/sync/ while
keeping POST@/repository//version/ while addressing the plug-in
involvement absence, results in a design where plug-in configuration
objects need to exist to be passed thru as parameters to those
endpoints and causing inconsistency with plug-in specific version
creation endpoints.
At the same time we are aware of a solution that addresses all three
issues at once by a design enhancement.
What would be the motivation to enumerate the issues separately?

>
>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald  wrote:
>>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>> Action endpoints are problematic.
>>
>> Example POST@/importers//sync/
>> They are non-RESTful and would make client code tightly coupled with the
>> server code.
>> These endpoints are inconsistent with the other parts of the REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not RESTful"
> perhaps and have the self-consistency bullet removed?

In my mind being RESTful implies consistency but I might be wrong.

>>
>> DRF is not being used as intended for action endpoints so we have to
>> implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?

for instance, custom action endpoints need custom documentation
workarounds; see https://github.com/pulp/pulp_file/pull/60
TL;DR: working around POST@/type/ that doesn't create an instance of the type.

>
>>
>> We don't have a convention for where plug-in-specific, custom repository
>> version creation endpoints.
>>
>> example POST@/api/v3//docker/add/
>> needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed in the
> schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?

The point here is to stress the absence of a good rule-of-a-thumb.
Currently POST@importers///sync/ and
POST@/repository//version/ create repository version.
Not being careful about a good design (by e.g preserving these core
endpoints), custom plug-in actions might end up being yet another
place where repository versions would be created.

>
>>
>> For direct repository version creation, plugins are not involved.
>>
>> validation correctness problem: https://pulp.plan.io/issues/3541
>> example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
> plugin writers but not all.

Yet a good design wouldn't stand in the way of plug-ins that won't
need the involvement by e.g not forcing a core endpoint to require
plug-in configuration to be passed thru.


Thanks,
milan

>
>>
>> We would like to get feedback on these issues being sound and worth
>> resolving before we resume particular solution discussion[1].
>>
>> Thanks,
>> Austin, Dennis, and Milan
>>
>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Brian Bouterse]

These are good problem statements. I didn't understand all of the aspects
of it, so I put some inline questions.

My overall question is: are these related problems? To share my answer to
this, I believe the first two problems are related and the third is
separate. The classic divide and conquor approach we could use here is to
confirm that the problems are unrelated and focus on resolving one of them
first.


On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald  wrote:

> Folks,
>
> Austin, Dennis, and Milan have identified the following issues with
> current Pulp3 REST API design:
>
>- Action endpoints are problematic.
>- Example POST@/importers//sync/
>   - They are non-RESTful and would make client code tightly coupled
>   with the server code.
>   - These endpoints are inconsistent with the other parts of the REST
>   API.
>
> Is self-consistency really a goal? I think it's a placeholder for
consistency for REST since the "rest" of the API is RESTful. After reading
parts of Roy Fielding's writeup of the definition of REST I believe "action
endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
are problematic" should be replaced with "Action endpoints are not RESTful"
perhaps and have the self-consistency bullet removed?

>
>- DRF is not being used as intended for action endpoints so we have to
>   implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?


>
>- We don't have a convention for where plug-in-specific, custom
>repository version creation endpoints.
>- example POST@/api/v3//docker/add/
>   - needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed in
the schema?

I think of ^ problem somewhat differently. Yes all urls need to be
discoverable (a REST property), but isn't it more of an issue that the urls
which produce repo versions can't be identified distinctly from any other
plugin-contributed url? To paraphrase this perspective: making a repo
version is strewn about throughout the API in random places which is a bad
user experience. Is that what is motivation url discovery?


>
>- For direct repository version creation, plugins are not involved.
>- validation correctness problem: https://pulp.plan.io/issues/3541
>   - example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
plugin writers but not all.


> We would like to get feedback on these issues being sound and worth
> resolving before we resume particular solution discussion[1].
>
> Thanks,
> Austin, Dennis, and Milan
>
> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Jeff Ortel]

On 04/09/2018 02:18 PM, Austin Macdonald wrote:

Folks,

Austin, Dennis, and Milan have identified the following issues with 
current Pulp3 REST API design:


  * Action endpoints are problematic.
  o Example POST@/importers//sync/
  o They are non-RESTful and would make client code tightly
coupled with the server code.
  o These endpoints are inconsistent with the other parts of the
REST API.
  o DRF is not being used as intended for action endpoints so we
have to implement extra code. (against the grain)


+1


  * We don't have a convention for where plug-in-specific, custom
repository version creation endpoints.
  o example POST@/api/v3//docker/add/
  o needs to be discoverable through the schema



+1


  * For direct repository version creation, plugins are not involved.
  o validation correctness problem:
https://pulp.plan.io/issues/3541

  o example: POST@/api/v3/repositories//versions/



Looks like half of the plugins will need to participate in creating 
repository versions (outside of sync). The API design should take a 
consistent approach to creating repository versions (/add/ _and_ /sync/).


We would like to get feedback on these issues being sound and worth 
resolving before we resume particular solution discussion[1].


Thanks,
Austin, Dennis, and Milan

[1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html 





___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Dana Walker]

I agree on these issues being sound and worth resolving.  I would much
prefer we maintain a truly RESTful API if possible. +1 resuming particular
solution discussion.

--Dana

Dana Walker

Associate Software Engineer

Red Hat




On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald  wrote:

> Folks,
>
> Austin, Dennis, and Milan have identified the following issues with
> current Pulp3 REST API design:
>
>- Action endpoints are problematic.
>- Example POST@/importers//sync/
>   - They are non-RESTful and would make client code tightly coupled
>   with the server code.
>   - These endpoints are inconsistent with the other parts of the REST
>   API.
>   - DRF is not being used as intended for action endpoints so we have
>   to implement extra code. (against the grain)
>- We don't have a convention for where plug-in-specific, custom
>repository version creation endpoints.
>- example POST@/api/v3//docker/add/
>   - needs to be discoverable through the schema
>- For direct repository version creation, plugins are not involved.
>- validation correctness problem: https://pulp.plan.io/issues/3541
>   - example: POST@/api/v3/repositories//versions/
>
> We would like to get feedback on these issues being sound and worth
> resolving before we resume particular solution discussion[1].
>
> Thanks,
> Austin, Dennis, and Milan
>
> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev