Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Brian Bouterse]

During the demo yesterday it was pointed out that 'cache_only' doesn't make
sense as a name anymore since Pulp itself isn't doing the caching. Here's
an issue to rename it with some naming suggestions:

https://pulp.plan.io/issues/4243

On Thu, Dec 6, 2018 at 3:26 PM Brian Bouterse  wrote:

> I've heard a lot of positive feedback on this idea and no objections. I
> made this story tracking this threads proposal specifically [0] and I made
> it a subtask of the lazy epic [1]. My next step to finish the streamer is
> to work on [0] so I've taken that as assigned. I'll reply to the list when
> the next round of PRs are posted and passing in Travis. Hopefully that
> should complete most of the work for the lazy epic [1].
>
> [0]: https://pulp.plan.io/issues/4239
> [1]: https://pulp.plan.io/issues/3693
>
> On Wed, Dec 5, 2018 at 4:07 PM Dennis Kliban  wrote:
>
>> Since no one is objecting, I'd like to see the PR[0] from @bmbouter
>> finished up and merged soon. I am updating the pass-through cache story[1]
>> to get it ready for implementation. I would like to remove all mention of
>> separate content app and streamer from the description.
>>
>>
>> [0] https://github.com/pulp/pulp/pull/3779
>> [1] https://pulp.plan.io/issues/3894
>>
>> On Tue, Dec 4, 2018 at 1:54 PM Tatiana Tereshchenko 
>> wrote:
>>
>>> +1 to merge
>>> +1 to have clear docs for plugin writers how to create their own content
>>> app
>>>
>>> On Mon, Dec 3, 2018 at 11:25 PM Dennis Kliban 
>>> wrote:
>>>
 It was pointed out on IRC that plugins that have to supply their own
 content app (such as docker) currently need to supply 2 implementations of
 it in order to support on-demand use cases. One using django and another
 using aiohttp.

 We should not burden plugin writers in such a way. We really have to
 make the proposed change.

 On Mon, Dec 3, 2018 at 3:24 PM Dana Walker  wrote:

> In light of the efficiency gains and lack of significant drawbacks,
> I'm +1 on this proposal.
>
> --Dana
>
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
>
> On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban 
> wrote:
>
>> I like the idea of combining the two applications for all the reasons
>> already outlined on this thread. The user experience is going to be
>> simplified by this change. However, I want to point out that it will also
>> alter the plugin writer experience. Plugin writers that want to have 
>> their
>> own content app will now need to provide it as a plugin for the content 
>> app
>> (which is not a Django project). We should be able to clearly document 
>> this
>> for plugin writers. pulp_docker plugin will need to adopt this change. 
>> For
>> that reason I'd like us to make a decision on this soon.
>>
>> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:
>>
>>> *BACKGROUND*
>>>
>>> The pulp3 content app and the streamer (in-progress) currently have
>>> a lot of duplicate code and functionality.  At the very least, I think
>>> there is a opportunity to refactor both and share code.  But, this would
>>> leave us with two components with significant overlap in functionality.
>>>
>>> The functionality exclusive to the content-app:
>>>   - Optionally delegate file serving to a web server. (Eg:
>>> mod_xsendfile).
>>>   - Optional redirect to the streamer.
>>>
>>> The functionality exclusive to the streamer:
>>>   - Using the Remote & RemoteArtifact to download the file and
>>> stream on demand.
>>>
>>> Not much difference which raises the question: "Why do we have
>>> both?"  I think the answer may be that we don't.
>>>
>>> *PROPOSAL*
>>>
>>> Let's pull the content-app out and merge it with the streamer.  The
>>> new content (app) would have *streamer* architecture &
>>> functionality.  When a requested artifact has not been downloaded, it 
>>> would
>>> download/streamed instead of REDIRECT.  This does mean that deployments 
>>> and
>>> development environments would need to run an additional service to 
>>> serve
>>> content.  The /pulp/content endpoint would be on a different port than 
>>> the
>>> API.  I see this separation as a healthy thing.  There is significant
>>> efficiency to be gained as well.  Let's start with eliminating the
>>> REDIRECTs.  Cutting the GET requests in half is a win for both the 
>>> client,
>>> the network and the Pulp web stack.  Next is database queries.  Since 
>>> both
>>> applications needed to perform many of the same queries, combining the
>>> applications will roughly cut them in half as well.  Since the streamer 
>>> is
>>> based on asyncio and so would the merged app.
>>>
>>> There are probably lots of 

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Brian Bouterse]

I've heard a lot of positive feedback on this idea and no objections. I
made this story tracking this threads proposal specifically [0] and I made
it a subtask of the lazy epic [1]. My next step to finish the streamer is
to work on [0] so I've taken that as assigned. I'll reply to the list when
the next round of PRs are posted and passing in Travis. Hopefully that
should complete most of the work for the lazy epic [1].

[0]: https://pulp.plan.io/issues/4239
[1]: https://pulp.plan.io/issues/3693

On Wed, Dec 5, 2018 at 4:07 PM Dennis Kliban  wrote:

> Since no one is objecting, I'd like to see the PR[0] from @bmbouter
> finished up and merged soon. I am updating the pass-through cache story[1]
> to get it ready for implementation. I would like to remove all mention of
> separate content app and streamer from the description.
>
>
> [0] https://github.com/pulp/pulp/pull/3779
> [1] https://pulp.plan.io/issues/3894
>
> On Tue, Dec 4, 2018 at 1:54 PM Tatiana Tereshchenko 
> wrote:
>
>> +1 to merge
>> +1 to have clear docs for plugin writers how to create their own content
>> app
>>
>> On Mon, Dec 3, 2018 at 11:25 PM Dennis Kliban  wrote:
>>
>>> It was pointed out on IRC that plugins that have to supply their own
>>> content app (such as docker) currently need to supply 2 implementations of
>>> it in order to support on-demand use cases. One using django and another
>>> using aiohttp.
>>>
>>> We should not burden plugin writers in such a way. We really have to
>>> make the proposed change.
>>>
>>> On Mon, Dec 3, 2018 at 3:24 PM Dana Walker  wrote:
>>>
 In light of the efficiency gains and lack of significant drawbacks, I'm
 +1 on this proposal.

 --Dana


 Dana Walker

 Associate Software Engineer

 Red Hat

 
 


 On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban 
 wrote:

> I like the idea of combining the two applications for all the reasons
> already outlined on this thread. The user experience is going to be
> simplified by this change. However, I want to point out that it will also
> alter the plugin writer experience. Plugin writers that want to have their
> own content app will now need to provide it as a plugin for the content 
> app
> (which is not a Django project). We should be able to clearly document 
> this
> for plugin writers. pulp_docker plugin will need to adopt this change. For
> that reason I'd like us to make a decision on this soon.
>
> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:
>
>> *BACKGROUND*
>>
>> The pulp3 content app and the streamer (in-progress) currently have a
>> lot of duplicate code and functionality.  At the very least, I think 
>> there
>> is a opportunity to refactor both and share code.  But, this would leave 
>> us
>> with two components with significant overlap in functionality.
>>
>> The functionality exclusive to the content-app:
>>   - Optionally delegate file serving to a web server. (Eg:
>> mod_xsendfile).
>>   - Optional redirect to the streamer.
>>
>> The functionality exclusive to the streamer:
>>   - Using the Remote & RemoteArtifact to download the file and stream
>> on demand.
>>
>> Not much difference which raises the question: "Why do we have
>> both?"  I think the answer may be that we don't.
>>
>> *PROPOSAL*
>>
>> Let's pull the content-app out and merge it with the streamer.  The
>> new content (app) would have *streamer* architecture &
>> functionality.  When a requested artifact has not been downloaded, it 
>> would
>> download/streamed instead of REDIRECT.  This does mean that deployments 
>> and
>> development environments would need to run an additional service to serve
>> content.  The /pulp/content endpoint would be on a different port than 
>> the
>> API.  I see this separation as a healthy thing.  There is significant
>> efficiency to be gained as well.  Let's start with eliminating the
>> REDIRECTs.  Cutting the GET requests in half is a win for both the 
>> client,
>> the network and the Pulp web stack.  Next is database queries.  Since 
>> both
>> applications needed to perform many of the same queries, combining the
>> applications will roughly cut them in half as well.  Since the streamer 
>> is
>> based on asyncio and so would the merged app.
>>
>> There are probably lots of other pros/cons I have not considered but
>> it seems relatively straight forward.
>>
>> I'm thinking the new content app/service would be named:
>> *pulp-content*.
>>
>> Thoughts?
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> 

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Dennis Kliban]

Since no one is objecting, I'd like to see the PR[0] from @bmbouter
finished up and merged soon. I am updating the pass-through cache story[1]
to get it ready for implementation. I would like to remove all mention of
separate content app and streamer from the description.


[0] https://github.com/pulp/pulp/pull/3779
[1] https://pulp.plan.io/issues/3894

On Tue, Dec 4, 2018 at 1:54 PM Tatiana Tereshchenko 
wrote:

> +1 to merge
> +1 to have clear docs for plugin writers how to create their own content
> app
>
> On Mon, Dec 3, 2018 at 11:25 PM Dennis Kliban  wrote:
>
>> It was pointed out on IRC that plugins that have to supply their own
>> content app (such as docker) currently need to supply 2 implementations of
>> it in order to support on-demand use cases. One using django and another
>> using aiohttp.
>>
>> We should not burden plugin writers in such a way. We really have to make
>> the proposed change.
>>
>> On Mon, Dec 3, 2018 at 3:24 PM Dana Walker  wrote:
>>
>>> In light of the efficiency gains and lack of significant drawbacks, I'm
>>> +1 on this proposal.
>>>
>>> --Dana
>>>
>>>
>>> Dana Walker
>>>
>>> Associate Software Engineer
>>>
>>> Red Hat
>>>
>>> 
>>> 
>>>
>>>
>>> On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban  wrote:
>>>
 I like the idea of combining the two applications for all the reasons
 already outlined on this thread. The user experience is going to be
 simplified by this change. However, I want to point out that it will also
 alter the plugin writer experience. Plugin writers that want to have their
 own content app will now need to provide it as a plugin for the content app
 (which is not a Django project). We should be able to clearly document this
 for plugin writers. pulp_docker plugin will need to adopt this change. For
 that reason I'd like us to make a decision on this soon.

 On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:

> *BACKGROUND*
>
> The pulp3 content app and the streamer (in-progress) currently have a
> lot of duplicate code and functionality.  At the very least, I think there
> is a opportunity to refactor both and share code.  But, this would leave 
> us
> with two components with significant overlap in functionality.
>
> The functionality exclusive to the content-app:
>   - Optionally delegate file serving to a web server. (Eg:
> mod_xsendfile).
>   - Optional redirect to the streamer.
>
> The functionality exclusive to the streamer:
>   - Using the Remote & RemoteArtifact to download the file and stream
> on demand.
>
> Not much difference which raises the question: "Why do we have both?"
> I think the answer may be that we don't.
>
> *PROPOSAL*
>
> Let's pull the content-app out and merge it with the streamer.  The
> new content (app) would have *streamer* architecture &
> functionality.  When a requested artifact has not been downloaded, it 
> would
> download/streamed instead of REDIRECT.  This does mean that deployments 
> and
> development environments would need to run an additional service to serve
> content.  The /pulp/content endpoint would be on a different port than the
> API.  I see this separation as a healthy thing.  There is significant
> efficiency to be gained as well.  Let's start with eliminating the
> REDIRECTs.  Cutting the GET requests in half is a win for both the client,
> the network and the Pulp web stack.  Next is database queries.  Since both
> applications needed to perform many of the same queries, combining the
> applications will roughly cut them in half as well.  Since the streamer is
> based on asyncio and so would the merged app.
>
> There are probably lots of other pros/cons I have not considered but
> it seems relatively straight forward.
>
> I'm thinking the new content app/service would be named:
> *pulp-content*.
>
> Thoughts?
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Tatiana Tereshchenko]

+1 to merge
+1 to have clear docs for plugin writers how to create their own content app

On Mon, Dec 3, 2018 at 11:25 PM Dennis Kliban  wrote:

> It was pointed out on IRC that plugins that have to supply their own
> content app (such as docker) currently need to supply 2 implementations of
> it in order to support on-demand use cases. One using django and another
> using aiohttp.
>
> We should not burden plugin writers in such a way. We really have to make
> the proposed change.
>
> On Mon, Dec 3, 2018 at 3:24 PM Dana Walker  wrote:
>
>> In light of the efficiency gains and lack of significant drawbacks, I'm
>> +1 on this proposal.
>>
>> --Dana
>>
>>
>> Dana Walker
>>
>> Associate Software Engineer
>>
>> Red Hat
>>
>> 
>> 
>>
>>
>> On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban  wrote:
>>
>>> I like the idea of combining the two applications for all the reasons
>>> already outlined on this thread. The user experience is going to be
>>> simplified by this change. However, I want to point out that it will also
>>> alter the plugin writer experience. Plugin writers that want to have their
>>> own content app will now need to provide it as a plugin for the content app
>>> (which is not a Django project). We should be able to clearly document this
>>> for plugin writers. pulp_docker plugin will need to adopt this change. For
>>> that reason I'd like us to make a decision on this soon.
>>>
>>> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:
>>>
 *BACKGROUND*

 The pulp3 content app and the streamer (in-progress) currently have a
 lot of duplicate code and functionality.  At the very least, I think there
 is a opportunity to refactor both and share code.  But, this would leave us
 with two components with significant overlap in functionality.

 The functionality exclusive to the content-app:
   - Optionally delegate file serving to a web server. (Eg:
 mod_xsendfile).
   - Optional redirect to the streamer.

 The functionality exclusive to the streamer:
   - Using the Remote & RemoteArtifact to download the file and stream
 on demand.

 Not much difference which raises the question: "Why do we have both?"
 I think the answer may be that we don't.

 *PROPOSAL*

 Let's pull the content-app out and merge it with the streamer.  The new
 content (app) would have *streamer* architecture & functionality.
 When a requested artifact has not been downloaded, it would
 download/streamed instead of REDIRECT.  This does mean that deployments and
 development environments would need to run an additional service to serve
 content.  The /pulp/content endpoint would be on a different port than the
 API.  I see this separation as a healthy thing.  There is significant
 efficiency to be gained as well.  Let's start with eliminating the
 REDIRECTs.  Cutting the GET requests in half is a win for both the client,
 the network and the Pulp web stack.  Next is database queries.  Since both
 applications needed to perform many of the same queries, combining the
 applications will roughly cut them in half as well.  Since the streamer is
 based on asyncio and so would the merged app.

 There are probably lots of other pros/cons I have not considered but it
 seems relatively straight forward.

 I'm thinking the new content app/service would be named: *pulp-content*
 .

 Thoughts?

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev

>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Dennis Kliban]

It was pointed out on IRC that plugins that have to supply their own
content app (such as docker) currently need to supply 2 implementations of
it in order to support on-demand use cases. One using django and another
using aiohttp.

We should not burden plugin writers in such a way. We really have to make
the proposed change.

On Mon, Dec 3, 2018 at 3:24 PM Dana Walker  wrote:

> In light of the efficiency gains and lack of significant drawbacks, I'm +1
> on this proposal.
>
> --Dana
>
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
>
> On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban  wrote:
>
>> I like the idea of combining the two applications for all the reasons
>> already outlined on this thread. The user experience is going to be
>> simplified by this change. However, I want to point out that it will also
>> alter the plugin writer experience. Plugin writers that want to have their
>> own content app will now need to provide it as a plugin for the content app
>> (which is not a Django project). We should be able to clearly document this
>> for plugin writers. pulp_docker plugin will need to adopt this change. For
>> that reason I'd like us to make a decision on this soon.
>>
>> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:
>>
>>> *BACKGROUND*
>>>
>>> The pulp3 content app and the streamer (in-progress) currently have a
>>> lot of duplicate code and functionality.  At the very least, I think there
>>> is a opportunity to refactor both and share code.  But, this would leave us
>>> with two components with significant overlap in functionality.
>>>
>>> The functionality exclusive to the content-app:
>>>   - Optionally delegate file serving to a web server. (Eg:
>>> mod_xsendfile).
>>>   - Optional redirect to the streamer.
>>>
>>> The functionality exclusive to the streamer:
>>>   - Using the Remote & RemoteArtifact to download the file and stream on
>>> demand.
>>>
>>> Not much difference which raises the question: "Why do we have both?"  I
>>> think the answer may be that we don't.
>>>
>>> *PROPOSAL*
>>>
>>> Let's pull the content-app out and merge it with the streamer.  The new
>>> content (app) would have *streamer* architecture & functionality.  When
>>> a requested artifact has not been downloaded, it would download/streamed
>>> instead of REDIRECT.  This does mean that deployments and development
>>> environments would need to run an additional service to serve content.  The
>>> /pulp/content endpoint would be on a different port than the API.  I see
>>> this separation as a healthy thing.  There is significant efficiency to be
>>> gained as well.  Let's start with eliminating the REDIRECTs.  Cutting the
>>> GET requests in half is a win for both the client, the network and the Pulp
>>> web stack.  Next is database queries.  Since both applications needed to
>>> perform many of the same queries, combining the applications will roughly
>>> cut them in half as well.  Since the streamer is based on asyncio and so
>>> would the merged app.
>>>
>>> There are probably lots of other pros/cons I have not considered but it
>>> seems relatively straight forward.
>>>
>>> I'm thinking the new content app/service would be named: *pulp-content*.
>>>
>>> Thoughts?
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Dana Walker]

In light of the efficiency gains and lack of significant drawbacks, I'm +1
on this proposal.

--Dana


Dana Walker

Associate Software Engineer

Red Hat





On Mon, Dec 3, 2018 at 2:40 PM Dennis Kliban  wrote:

> I like the idea of combining the two applications for all the reasons
> already outlined on this thread. The user experience is going to be
> simplified by this change. However, I want to point out that it will also
> alter the plugin writer experience. Plugin writers that want to have their
> own content app will now need to provide it as a plugin for the content app
> (which is not a Django project). We should be able to clearly document this
> for plugin writers. pulp_docker plugin will need to adopt this change. For
> that reason I'd like us to make a decision on this soon.
>
> On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:
>
>> *BACKGROUND*
>>
>> The pulp3 content app and the streamer (in-progress) currently have a lot
>> of duplicate code and functionality.  At the very least, I think there is a
>> opportunity to refactor both and share code.  But, this would leave us with
>> two components with significant overlap in functionality.
>>
>> The functionality exclusive to the content-app:
>>   - Optionally delegate file serving to a web server. (Eg: mod_xsendfile).
>>   - Optional redirect to the streamer.
>>
>> The functionality exclusive to the streamer:
>>   - Using the Remote & RemoteArtifact to download the file and stream on
>> demand.
>>
>> Not much difference which raises the question: "Why do we have both?"  I
>> think the answer may be that we don't.
>>
>> *PROPOSAL*
>>
>> Let's pull the content-app out and merge it with the streamer.  The new
>> content (app) would have *streamer* architecture & functionality.  When
>> a requested artifact has not been downloaded, it would download/streamed
>> instead of REDIRECT.  This does mean that deployments and development
>> environments would need to run an additional service to serve content.  The
>> /pulp/content endpoint would be on a different port than the API.  I see
>> this separation as a healthy thing.  There is significant efficiency to be
>> gained as well.  Let's start with eliminating the REDIRECTs.  Cutting the
>> GET requests in half is a win for both the client, the network and the Pulp
>> web stack.  Next is database queries.  Since both applications needed to
>> perform many of the same queries, combining the applications will roughly
>> cut them in half as well.  Since the streamer is based on asyncio and so
>> would the merged app.
>>
>> There are probably lots of other pros/cons I have not considered but it
>> seems relatively straight forward.
>>
>> I'm thinking the new content app/service would be named: *pulp-content*.
>>
>> Thoughts?
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Dennis Kliban]

I like the idea of combining the two applications for all the reasons
already outlined on this thread. The user experience is going to be
simplified by this change. However, I want to point out that it will also
alter the plugin writer experience. Plugin writers that want to have their
own content app will now need to provide it as a plugin for the content app
(which is not a Django project). We should be able to clearly document this
for plugin writers. pulp_docker plugin will need to adopt this change. For
that reason I'd like us to make a decision on this soon.

On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:

> *BACKGROUND*
>
> The pulp3 content app and the streamer (in-progress) currently have a lot
> of duplicate code and functionality.  At the very least, I think there is a
> opportunity to refactor both and share code.  But, this would leave us with
> two components with significant overlap in functionality.
>
> The functionality exclusive to the content-app:
>   - Optionally delegate file serving to a web server. (Eg: mod_xsendfile).
>   - Optional redirect to the streamer.
>
> The functionality exclusive to the streamer:
>   - Using the Remote & RemoteArtifact to download the file and stream on
> demand.
>
> Not much difference which raises the question: "Why do we have both?"  I
> think the answer may be that we don't.
>
> *PROPOSAL*
>
> Let's pull the content-app out and merge it with the streamer.  The new
> content (app) would have *streamer* architecture & functionality.  When a
> requested artifact has not been downloaded, it would download/streamed
> instead of REDIRECT.  This does mean that deployments and development
> environments would need to run an additional service to serve content.  The
> /pulp/content endpoint would be on a different port than the API.  I see
> this separation as a healthy thing.  There is significant efficiency to be
> gained as well.  Let's start with eliminating the REDIRECTs.  Cutting the
> GET requests in half is a win for both the client, the network and the Pulp
> web stack.  Next is database queries.  Since both applications needed to
> perform many of the same queries, combining the applications will roughly
> cut them in half as well.  Since the streamer is based on asyncio and so
> would the merged app.
>
> There are probably lots of other pros/cons I have not considered but it
> seems relatively straight forward.
>
> I'm thinking the new content app/service would be named: *pulp-content*.
>
> Thoughts?
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Proposal: merge the content-app & streamer

[Brian Bouterse]

Jeff mentioned this to me yesterday, and this decision greatly affects the
remaining work of the streamer. Jeff and I thought a prototype also would
be easy and help make the idea more concrete. Here is some code you can try
with this idea. This prototype took about an hour, so we can throw it away
or delete it if we want.

This is the gist of the pulpcore changes (no docs or travis updates):
https://github.com/pulp/pulp/pull/3779
Use this branch of the streamer (updated for settings changes from ^):
https://github.com/bmbouter/pulp_streamer/tree/streamer-replaces-content-app

In addition to the great benefits Jeff outlined, in my testing I can see
the aiohttp based streamer is generally a lot faster in handling requests
than Django which could be big for our performance. Also, it simplifies the
settings greatly since we don't need redirect configs anymore (see the PR).

More thoughts would be great. The lazy work is blocked until we're able to
pick a direction here.

Thanks!

On Fri, Nov 30, 2018 at 4:59 PM Jeff Ortel  wrote:

> *BACKGROUND*
>
> The pulp3 content app and the streamer (in-progress) currently have a lot
> of duplicate code and functionality.  At the very least, I think there is a
> opportunity to refactor both and share code.  But, this would leave us with
> two components with significant overlap in functionality.
>
> The functionality exclusive to the content-app:
>   - Optionally delegate file serving to a web server. (Eg: mod_xsendfile).
>   - Optional redirect to the streamer.
>
> The functionality exclusive to the streamer:
>   - Using the Remote & RemoteArtifact to download the file and stream on
> demand.
>
> Not much difference which raises the question: "Why do we have both?"  I
> think the answer may be that we don't.
>
> *PROPOSAL*
>
> Let's pull the content-app out and merge it with the streamer.  The new
> content (app) would have *streamer* architecture & functionality.  When a
> requested artifact has not been downloaded, it would download/streamed
> instead of REDIRECT.  This does mean that deployments and development
> environments would need to run an additional service to serve content.  The
> /pulp/content endpoint would be on a different port than the API.  I see
> this separation as a healthy thing.  There is significant efficiency to be
> gained as well.  Let's start with eliminating the REDIRECTs.  Cutting the
> GET requests in half is a win for both the client, the network and the Pulp
> web stack.  Next is database queries.  Since both applications needed to
> perform many of the same queries, combining the applications will roughly
> cut them in half as well.  Since the streamer is based on asyncio and so
> would the merged app.
>
> There are probably lots of other pros/cons I have not considered but it
> seems relatively straight forward.
>
> I'm thinking the new content app/service would be named: *pulp-content*.
>
> Thoughts?
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev