Re: [Pulp-dev] file repository type

[Brian Bouterse]

Yes the git syncing should probably be in a version control plugin that
syncs and publishes different VCS protocols like git.

+1 to the file plugin doing all of those use cases. Anyone want to file any
stories? If we can plan it some, maybe someone will do it.
https://pulp.plan.io/projects/pulp_file/issues/new

I have ideas about the file plugin, but I don't actually use it.

On Thu, Jun 15, 2017 at 2:37 PM, Eric Helms  wrote:

> A few suggestions from my own heavy use of the current "plugin" (and I may
> repeat some previously said ideas):
>
>  1) consider supporting syncing from a few well known types: ftp, apache
> static file server
>  2) Support "uploading" a file from a url
>  3) Look at what wget does and recursively sync down whatever is over at
> location X
>  4) Use name and path as the "ID" for a file to allow re-uploads of the
> same file name
>
> I agree that syncing Git would be cool, but should be it's own entity
> since Git is a known "protocol". The wget way is interesting in that I used
> that to sync debian to a file repo:
>
>  1) I synced via recursive wget a debian repository
>  2) I used the script at https://gist.github.com/ehelms/
> 3fd956ee887db3d7bac20b29efa3dd51 to generate a recursive (with full file
> path) PULP_MANIFEST
>  3) I then synced that to fully mirror the repository
>
> Steps 1-3 seem common enough that I shouldn't have to do it and/or waste
> storage space just to replicate a potentially large file location.
>
> Eric
>
> On Tue, Jun 6, 2017 at 6:02 PM, Tom McKay  wrote:
>
>> As a user, the standard is "wget". In foreman I entered a url to a raw
>> github file... and it failed (obscurely). While one could argue for better
>> error messaging, I would argue that I was pretty explicit in indicating I
>> wanted the results returned from the url. When I sync or resync the
>> content, just go wget it again and stuff it into the repo. Yes, it's a repo
>> of one but in foreman I make "products" which are collections of repos.
>> (Agreed that my next ask would be to "sync all the files listed at a url. :)
>>
>>
>>
>>
>> On Tue, Jun 6, 2017 at 5:03 PM, Michael Hrivnak 
>> wrote:
>>
>>> The key is in discovery of files. There's no standard we know of that
>>> would be helpful, which is why we depend on a PULP_MANIFEST to tell us what
>>> files are available. Maybe there are some cases when a particular service
>>> has a well-defined way of expressing what files are available, in which
>>> case we could instead support that as the discovery mechanism. It's a good
>>> question about whether that would be an addition to the pulp_file plugin,
>>> or a separate plugin. It probably needs to be evaluated case-by-case.
>>>
>>> As one related example, the Pulp 2 puppet importer supports both the
>>> forge API and a PULP_MANIFEST for discovery of available puppet modules on
>>> a remote server.
>>>
>>> On Tue, Jun 6, 2017 at 4:46 PM, Brian Bouterse 
>>> wrote:
>>>
 What if the pulp_file plugin (the pulp3 one) would support syncing
 files directly from remotes without a manifest? Like syncing from a remote
 webserver or version control system. Maybe that would be a VCS plugin, not
 the file_plugin, but regardless, other software similar to Pulp supports
 [0] that use case.

 [0]: http://bit.ly/2s1pCmF

 On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak 
 wrote:

> You do not need to create a manifest in order to upload. You only need
> one in order to sync. Here is documentation about upload:
>
> http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
> .html#uploading-isos-to-a-repository
>
> You can also create the manifest file in 3-4 lines of bash. Maybe it
> would be useful for Pulp to include a small script that can generate the
> manifest.
>
> That said, you can always upload your files to a Pulp repo, and
> publish it, which will cause Pulp to create a new manifest for you.
>
> On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay 
> wrote:
>
>> I'd like to sync individual files and folders (recursively?) into
>> pulp via foreman. Apparently, though, I need to create a pulp manifest in
>> order to upload. Is this necessary or can pulp do it for me? Or should I
>> build this into foreman? @ehelms provided a gist of the steps[1] and it
>> seems pretty basic and something pulp could be doing itself.
>>
>> My use case is that I wanted to import a build file for openshift
>> from github, then promote that file (and other artifacts) through 
>> foreman's
>> life cycle environments. A lot of demos and tutorials for openshift start
>> from git and other non-local components.
>>
>> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>>
>> ___

Re: [Pulp-dev] file repository type

[Eric Helms]

A few suggestions from my own heavy use of the current "plugin" (and I may
repeat some previously said ideas):

 1) consider supporting syncing from a few well known types: ftp, apache
static file server
 2) Support "uploading" a file from a url
 3) Look at what wget does and recursively sync down whatever is over at
location X
 4) Use name and path as the "ID" for a file to allow re-uploads of the
same file name

I agree that syncing Git would be cool, but should be it's own entity since
Git is a known "protocol". The wget way is interesting in that I used that
to sync debian to a file repo:

 1) I synced via recursive wget a debian repository
 2) I used the script at
https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51 to generate
a recursive (with full file path) PULP_MANIFEST
 3) I then synced that to fully mirror the repository

Steps 1-3 seem common enough that I shouldn't have to do it and/or waste
storage space just to replicate a potentially large file location.

Eric

On Tue, Jun 6, 2017 at 6:02 PM, Tom McKay  wrote:

> As a user, the standard is "wget". In foreman I entered a url to a raw
> github file... and it failed (obscurely). While one could argue for better
> error messaging, I would argue that I was pretty explicit in indicating I
> wanted the results returned from the url. When I sync or resync the
> content, just go wget it again and stuff it into the repo. Yes, it's a repo
> of one but in foreman I make "products" which are collections of repos.
> (Agreed that my next ask would be to "sync all the files listed at a url. :)
>
>
>
>
> On Tue, Jun 6, 2017 at 5:03 PM, Michael Hrivnak 
> wrote:
>
>> The key is in discovery of files. There's no standard we know of that
>> would be helpful, which is why we depend on a PULP_MANIFEST to tell us what
>> files are available. Maybe there are some cases when a particular service
>> has a well-defined way of expressing what files are available, in which
>> case we could instead support that as the discovery mechanism. It's a good
>> question about whether that would be an addition to the pulp_file plugin,
>> or a separate plugin. It probably needs to be evaluated case-by-case.
>>
>> As one related example, the Pulp 2 puppet importer supports both the
>> forge API and a PULP_MANIFEST for discovery of available puppet modules on
>> a remote server.
>>
>> On Tue, Jun 6, 2017 at 4:46 PM, Brian Bouterse 
>> wrote:
>>
>>> What if the pulp_file plugin (the pulp3 one) would support syncing files
>>> directly from remotes without a manifest? Like syncing from a remote
>>> webserver or version control system. Maybe that would be a VCS plugin, not
>>> the file_plugin, but regardless, other software similar to Pulp supports
>>> [0] that use case.
>>>
>>> [0]: http://bit.ly/2s1pCmF
>>>
>>> On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak 
>>> wrote:
>>>
 You do not need to create a manifest in order to upload. You only need
 one in order to sync. Here is documentation about upload:

 http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
 .html#uploading-isos-to-a-repository

 You can also create the manifest file in 3-4 lines of bash. Maybe it
 would be useful for Pulp to include a small script that can generate the
 manifest.

 That said, you can always upload your files to a Pulp repo, and publish
 it, which will cause Pulp to create a new manifest for you.

 On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay 
 wrote:

> I'd like to sync individual files and folders (recursively?) into pulp
> via foreman. Apparently, though, I need to create a pulp manifest in order
> to upload. Is this necessary or can pulp do it for me? Or should I build
> this into foreman? @ehelms provided a gist of the steps[1] and it seems
> pretty basic and something pulp could be doing itself.
>
> My use case is that I wanted to import a build file for openshift from
> github, then promote that file (and other artifacts) through foreman's 
> life
> cycle environments. A lot of demos and tutorials for openshift start from
> git and other non-local components.
>
> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>


 --

 Michael Hrivnak

 Principal Software Engineer, RHCE

 Red Hat

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>
>>
>> --
>>
>> Michael Hrivnak
>>
>> Principal Software Engineer, RHCE
>>
>> Red Hat
>>
>
>
> ___
> Pulp-dev mailing list
> 

Re: [Pulp-dev] file repository type

[Kodiak Firesmith]

Forgot to reply-all as usual.

On Jun 6, 2017 5:10 PM, wrote:

I thought I had an RFE in for this a couple years ago.  It would be very
nice to do non-manifest syncs against non-pulp remote sources.  Especially
if it supported the concept of recursive folder/file syncs.

On Jun 6, 2017 4:47 PM, "Brian Bouterse"  wrote:

What if the pulp_file plugin (the pulp3 one) would support syncing files
directly from remotes without a manifest? Like syncing from a remote
webserver or version control system. Maybe that would be a VCS plugin, not
the file_plugin, but regardless, other software similar to Pulp supports
[0] that use case.

[0]: http://bit.ly/2s1pCmF

On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak  wrote:

> You do not need to create a manifest in order to upload. You only need one
> in order to sync. Here is documentation about upload:
>
> http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
> .html#uploading-isos-to-a-repository
>
> You can also create the manifest file in 3-4 lines of bash. Maybe it would
> be useful for Pulp to include a small script that can generate the manifest.
>
> That said, you can always upload your files to a Pulp repo, and publish
> it, which will cause Pulp to create a new manifest for you.
>
> On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay  wrote:
>
>> I'd like to sync individual files and folders (recursively?) into pulp
>> via foreman. Apparently, though, I need to create a pulp manifest in order
>> to upload. Is this necessary or can pulp do it for me? Or should I build
>> this into foreman? @ehelms provided a gist of the steps[1] and it seems
>> pretty basic and something pulp could be doing itself.
>>
>> My use case is that I wanted to import a build file for openshift from
>> github, then promote that file (and other artifacts) through foreman's life
>> cycle environments. A lot of demos and tutorials for openshift start from
>> git and other non-local components.
>>
>> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
>
> --
>
> Michael Hrivnak
>
> Principal Software Engineer, RHCE
>
> Red Hat
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] file repository type

[Michael Hrivnak]

The key is in discovery of files. There's no standard we know of that would
be helpful, which is why we depend on a PULP_MANIFEST to tell us what files
are available. Maybe there are some cases when a particular service has a
well-defined way of expressing what files are available, in which case we
could instead support that as the discovery mechanism. It's a good question
about whether that would be an addition to the pulp_file plugin, or a
separate plugin. It probably needs to be evaluated case-by-case.

As one related example, the Pulp 2 puppet importer supports both the forge
API and a PULP_MANIFEST for discovery of available puppet modules on a
remote server.

On Tue, Jun 6, 2017 at 4:46 PM, Brian Bouterse  wrote:

> What if the pulp_file plugin (the pulp3 one) would support syncing files
> directly from remotes without a manifest? Like syncing from a remote
> webserver or version control system. Maybe that would be a VCS plugin, not
> the file_plugin, but regardless, other software similar to Pulp supports
> [0] that use case.
>
> [0]: http://bit.ly/2s1pCmF
>
> On Mon, Jun 5, 2017 at 1:10 PM, Michael Hrivnak 
> wrote:
>
>> You do not need to create a manifest in order to upload. You only need
>> one in order to sync. Here is documentation about upload:
>>
>> http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos
>> .html#uploading-isos-to-a-repository
>>
>> You can also create the manifest file in 3-4 lines of bash. Maybe it
>> would be useful for Pulp to include a small script that can generate the
>> manifest.
>>
>> That said, you can always upload your files to a Pulp repo, and publish
>> it, which will cause Pulp to create a new manifest for you.
>>
>> On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay  wrote:
>>
>>> I'd like to sync individual files and folders (recursively?) into pulp
>>> via foreman. Apparently, though, I need to create a pulp manifest in order
>>> to upload. Is this necessary or can pulp do it for me? Or should I build
>>> this into foreman? @ehelms provided a gist of the steps[1] and it seems
>>> pretty basic and something pulp could be doing itself.
>>>
>>> My use case is that I wanted to import a build file for openshift from
>>> github, then promote that file (and other artifacts) through foreman's life
>>> cycle environments. A lot of demos and tutorials for openshift start from
>>> git and other non-local components.
>>>
>>> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>>
>> --
>>
>> Michael Hrivnak
>>
>> Principal Software Engineer, RHCE
>>
>> Red Hat
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>


-- 

Michael Hrivnak

Principal Software Engineer, RHCE

Red Hat
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] file repository type

[Michael Hrivnak]

You do not need to create a manifest in order to upload. You only need one
in order to sync. Here is documentation about upload:

http://docs.pulpproject.org/plugins/pulp_rpm/user-guide/isos.html#uploading-isos-to-a-repository

You can also create the manifest file in 3-4 lines of bash. Maybe it would
be useful for Pulp to include a small script that can generate the manifest.

That said, you can always upload your files to a Pulp repo, and publish it,
which will cause Pulp to create a new manifest for you.

On Mon, Jun 5, 2017 at 6:41 AM, Tom McKay  wrote:

> I'd like to sync individual files and folders (recursively?) into pulp via
> foreman. Apparently, though, I need to create a pulp manifest in order to
> upload. Is this necessary or can pulp do it for me? Or should I build this
> into foreman? @ehelms provided a gist of the steps[1] and it seems pretty
> basic and something pulp could be doing itself.
>
> My use case is that I wanted to import a build file for openshift from
> github, then promote that file (and other artifacts) through foreman's life
> cycle environments. A lot of demos and tutorials for openshift start from
> git and other non-local components.
>
> [1] https://gist.github.com/ehelms/3fd956ee887db3d7bac20b29efa3dd51
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>


-- 

Michael Hrivnak

Principal Software Engineer, RHCE

Red Hat
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev