[Pulp-dev] Farewell party

[Milan Kovacik]

Hey guys,

I'm just about to leave but I'd really like to thank You all for having me
--- I had a great time!
I'm staying in Brno --- let me know when You're around ;)

Cheers,
milan

https://www.linkedin.com/in/milan-kovacik-261b5a10/
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 2 depsolving module errata

[Milan Kovacik]

I see, yeah, I'm a bit vague; should be like the more detail is specified
by by a specific treat requirement, the fewer options should match.
If there's a unit requiring "module(nodejs)" and there are modules
providing multiple streams of the nodejs content in the (source) repository
--- "module(nodejs:6), module(nodejs:8)" then Pulp may pick any of the
streams.
If a unit requires particular stream then the stream should be respected by
Pulp too. Ditto for the architecture.

--
milan

On Wed, Dec 5, 2018 at 2:46 PM Robin Chan  wrote:

> I have a question.
> Can you clarify the wording of "that particular module should be copied"
> in the last 3 bullet use cases? Perhaps a use case? To me same wording
> implies same behavior - or perhaps I'm not getting the distinction.  Thanks!
>
>
> On Wed, Dec 5, 2018 at 3:28 AM Milan Kovacik  wrote:
>
>>
>> Robin, I think you're right, we should include the folks.
>>
>> --
>> milan
>>
>> -- Forwarded message -
>> From: Robin Chan 
>> Date: Wed, Dec 5, 2018 at 1:02 AM
>> Subject: Re: Pulp 2 depsolving module errata
>> To: Milan Kovacik 
>> Cc: Daniel Alley , Kersom 
>>
>>
>> Can we continue convo in pull-dev? Feel like these use cases need to live
>> somewhere not email & other may have some input/interest.
>>
>> On Tue, Dec 4, 2018 at 5:03 PM Milan Kovacik  wrote:
>>
>>> Hi Kersom,
>>>
>>> I don't fully understand all the usecases yet but let me try to put some
>>> down here:
>>> * a recursive copy of a module pulls over all module artifacts
>>> * a recursive copy of a module pulls over all modular dependencies, the
>>> other modules this module depends on
>>> * a recursive copy of an artifact RPM pulls over all units the artifact
>>> may depend on
>>> * if a unit depends on a particular module:stream, that particular
>>> module should be copied
>>> * if a unit depends on a particular module:stream:version, that
>>> particular module should be copied
>>> * if a unit depends on a particular module:stream.architecture, then
>>> again, that particular module should be copied
>>>
>>> Modules behave a bit like repositories --- the consumer is supposed to
>>> enable a module and this should make DNF to prefer content from the module
>>> over the "ursine" RPMs that provide the same treats or cause a conflict
>>> with the modular RPMs (artifacts). To make the situation trickier, an
>>> ursine RPM can (afaik) depend on a module (stream) and modules can depend
>>> on other modules and the modular RPMs can depend on ursine RPMs. This has
>>> some edge-case implications when it comes to the content copy itself:
>>> * if there is an ursine RPM in a target repository and even if that
>>> ursine RPM has a newer version than a modular RPM providing the same treat
>>> (such as /bin/sh) and while that module is being copied, the modular
>>> (outdated) RPM should be "pulled" together with the module
>>> * if there are multiple modules with the same name that match the user
>>> copy query, every module should be copied (including their dependencies),
>>> even though the modules would conflict each other on a consumer system ---
>>> one can't enable at the same time multiple streams of the same module
>>>
>>> More specific to Pulp, the modular repositories are expected not to be
>>> closed on content dependencies; if this happens, Pulp shouldn't fail but
>>> instead, it should copy as many of the dependencies as possible. Pulp
>>> should not regress with modules in this regard i(n case we fix
>>> https://pulp.plan.io/issues/4152).
>>>
>>> Further more, we might end up with two algorithms that resolve the
>>> dependencies: a conservative one, that tries it's best to avoid
>>> unintentional "updates" in the target  repository, such as we introduced in
>>> https://pulp.plan.io/issues/2478 and another approach, that is more
>>> greedy and copies everything that provides particular treat (a content unit
>>> might depend on).
>>>
>>> Right now I've been just trying to experiment with the modular
>>> dependency solving on top on the Issue #4152 fix
>>> https://github.com/pulp/pulp_rpm/pull/1226, trying to figure out how to
>>> express these concerns in the libsolv terms while making sure the the two
>>> algorithms in the patch wouldn't break with modular content. I've got some
>>> progress but it's still in an early

[Pulp-dev] Fwd: Pulp 2 depsolving module errata

[Milan Kovacik]

Robin, I think you're right, we should include the folks.

--
milan

-- Forwarded message -
From: Robin Chan 
Date: Wed, Dec 5, 2018 at 1:02 AM
Subject: Re: Pulp 2 depsolving module errata
To: Milan Kovacik 
Cc: Daniel Alley , Kersom 


Can we continue convo in pull-dev? Feel like these use cases need to live
somewhere not email & other may have some input/interest.

On Tue, Dec 4, 2018 at 5:03 PM Milan Kovacik  wrote:

> Hi Kersom,
>
> I don't fully understand all the usecases yet but let me try to put some
> down here:
> * a recursive copy of a module pulls over all module artifacts
> * a recursive copy of a module pulls over all modular dependencies, the
> other modules this module depends on
> * a recursive copy of an artifact RPM pulls over all units the artifact
> may depend on
> * if a unit depends on a particular module:stream, that particular module
> should be copied
> * if a unit depends on a particular module:stream:version, that particular
> module should be copied
> * if a unit depends on a particular module:stream.architecture, then
> again, that particular module should be copied
>
> Modules behave a bit like repositories --- the consumer is supposed to
> enable a module and this should make DNF to prefer content from the module
> over the "ursine" RPMs that provide the same treats or cause a conflict
> with the modular RPMs (artifacts). To make the situation trickier, an
> ursine RPM can (afaik) depend on a module (stream) and modules can depend
> on other modules and the modular RPMs can depend on ursine RPMs. This has
> some edge-case implications when it comes to the content copy itself:
> * if there is an ursine RPM in a target repository and even if that ursine
> RPM has a newer version than a modular RPM providing the same treat (such
> as /bin/sh) and while that module is being copied, the modular (outdated)
> RPM should be "pulled" together with the module
> * if there are multiple modules with the same name that match the user
> copy query, every module should be copied (including their dependencies),
> even though the modules would conflict each other on a consumer system ---
> one can't enable at the same time multiple streams of the same module
>
> More specific to Pulp, the modular repositories are expected not to be
> closed on content dependencies; if this happens, Pulp shouldn't fail but
> instead, it should copy as many of the dependencies as possible. Pulp
> should not regress with modules in this regard i(n case we fix
> https://pulp.plan.io/issues/4152).
>
> Further more, we might end up with two algorithms that resolve the
> dependencies: a conservative one, that tries it's best to avoid
> unintentional "updates" in the target  repository, such as we introduced in
> https://pulp.plan.io/issues/2478 and another approach, that is more
> greedy and copies everything that provides particular treat (a content unit
> might depend on).
>
> Right now I've been just trying to experiment with the modular dependency
> solving on top on the Issue #4152 fix
> https://github.com/pulp/pulp_rpm/pull/1226, trying to figure out how to
> express these concerns in the libsolv terms while making sure the the two
> algorithms in the patch wouldn't break with modular content. I've got some
> progress but it's still in an early stage:
> https://github.com/dparalen/pulp_rpm/commit/43ae38a4ea2a2a843a42cc993e88cd3bf480ee9b
>
> Actually, we never groomed the modular depsolving story
> https://pulp.plan.io/issues/3740 so please take the usecases listed here
> with a grain of salt.
>
> Cheers,
> milan
>
> On Tue, Dec 4, 2018 at 9:54 PM Kersom  wrote:
>
>> Hi Milan,
>>
>> If there is any feature/issue related to depsolving that should be tested
>> or that QE should be aware of it, please , file test cases on
>> pulp.plan.io.
>>
>> Then we will be able to understand the scenarios and clarify doubts with
>> you.
>>
>> Regards,
>>
>> Kersom
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] To integrate Fus or not to....

[Milan Kovacik]

On Fri, Oct 12, 2018 at 7:11 PM Jeff Ortel  wrote:

>
>
> On 10/12/2018 11:37 AM, Milan Kovacik wrote:
>
>
>
> On Fri, Oct 12, 2018 at 5:17 PM Jeff Ortel  wrote:
>
>>
>>
>> On 10/12/2018 09:53 AM, Milan Kovacik wrote:
>>
>>
>>
>> On Fri, Oct 12, 2018 at 3:59 PM Jeff Ortel  wrote:
>>
>>>
>>>
>>> On 10/10/2018 08:59 AM, Milan Kovacik wrote:
>>>
>>> ...that might be the question we should ask ourselves once again when it
>>> comes to recursive copying of units between repositories.
>>>
>>> I'd like to poll folks opinions about the possibilities that we may have
>>> when it comes to integrating third party solvers in Pulp. My yesterday's
>>> chat with the #fedora-modularity folks about us integrating the Fus[1]
>>> solver in order to reuse the Fus algorithm ran into a couple of bumps:
>>>
>>> * it would be laborous to create a programmatic Python API between Fus
>>> and Pulp because we can't directly use the libsolv thingies (pools,
>>> solvables and friends) in such an API because Fus is written utilizing
>>> GObject, which is incompatible with Swig, which in turn is used in libsolv
>>> to expose the python bindings. One would have to either re-wrap libsolv
>>> code in Fus to work with pygobject or submit PRs against libsolv to support
>>> GObject introspection. I dunno the details of either approach (yet) but
>>> from the sad faces on the IRC and the Fus PR[1] it seemed like a lot of
>>> work but it's still an option
>>>
>>> * we still should be able to integrate thru a pipe into Fus, that would
>>> make it possible to dump modular and ursine metadata into Fus to perform
>>> the dependency solving in a separate subprocess. We should probably
>>> re-check the reasons behind our previous decision not to do the same with
>>> DNF[2].
>>>
>>>
>>> How is Integration with Fus via pipe (CLI) easier than with gobject?
>>> Either way, you "can't directly use the libsolv thingies (pools, solvables
>>> and friends)".  Right?  What am I missing?
>>>
>>>
>> Right, a publish-like operation would be required every time, for all
>> repositories involved in the copy to dump the metadata to the pipe(s);
>> sample of this interface is can be found in Pungi:
>> https://pagure.io/pungi/blob/master/f/pungi/wrappers/fus.py the "query"
>> is passed thru command line.
>> I just learnt Fedora will keep modules and their ursine deps in separate
>> repos, so the source repo won't necessarily be closed on dependencies thus
>> multiple source repos would be needed.
>>
>>
>> This be done using the Fus gobject interface as well?
>>
>
> we'd just dump the XML (and YAML) metadata and run: fus --repo
> source1,1,/path/to/pipe1 --repo source2,2,/path/to/pipe2 --repo
> target,system,/path/to/target_pipe  "module(walrus)" "penguin:1-2.3" etc
> then parse the textual output of fus such as:
>
>
> Can't this ^ be done with Fus through gobject as well and instead of
> parsing textual output, inspect the objects returned?
>
>
To avoid having to search for the resolved units (in Mongo), we have to
maintain a mapping[3] between the solver namespace and Pulp to be able to
refer back to the Pulp units.
Optimizing just the parsing wouldn't have much benefit;  we'd best benefit
from an API that would allow Pulp to have a reference to the solver unit
representation.
Currently we do this thru the libsolv.i[4] bindings therefore the Fus
(Py)GObject interface would have to expose a similar (hopefully smaller)
interface.
This still seemed a little bit too much work last time I checked with Fus
folks on #fedora-modularity.
Note please also that the GObject way won't allow for flexibility in the
solving algorithm[5] itself should Pulp require some adjustments in the
future.


[3]
https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/importers/yum/pulp_solv.py#L243,#L244
[4] https://github.com/openSUSE/libsolv/blob/master/bindings/solv.i
[5] https://github.com/fedora-modularity/fus/blob/master/fus.c#L955

>
> # ---%>-
>   - nothing provides policycoreutils-python-utils needed by
> container-selinux-2:2.69-3.git452b90d.module_2040+0e96cf1b.noarch
>
> Problem 1 / 1:
>   - conflicting requests
>   - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by
> atomic-1.22.1-2.module_1637+1872e86a.x86_64
>   - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by

Re: [Pulp-dev] To integrate Fus or not to....

[Milan Kovacik]

On Fri, Oct 12, 2018 at 5:17 PM Jeff Ortel  wrote:

>
>
> On 10/12/2018 09:53 AM, Milan Kovacik wrote:
>
>
>
> On Fri, Oct 12, 2018 at 3:59 PM Jeff Ortel  wrote:
>
>>
>>
>> On 10/10/2018 08:59 AM, Milan Kovacik wrote:
>>
>> ...that might be the question we should ask ourselves once again when it
>> comes to recursive copying of units between repositories.
>>
>> I'd like to poll folks opinions about the possibilities that we may have
>> when it comes to integrating third party solvers in Pulp. My yesterday's
>> chat with the #fedora-modularity folks about us integrating the Fus[1]
>> solver in order to reuse the Fus algorithm ran into a couple of bumps:
>>
>> * it would be laborous to create a programmatic Python API between Fus
>> and Pulp because we can't directly use the libsolv thingies (pools,
>> solvables and friends) in such an API because Fus is written utilizing
>> GObject, which is incompatible with Swig, which in turn is used in libsolv
>> to expose the python bindings. One would have to either re-wrap libsolv
>> code in Fus to work with pygobject or submit PRs against libsolv to support
>> GObject introspection. I dunno the details of either approach (yet) but
>> from the sad faces on the IRC and the Fus PR[1] it seemed like a lot of
>> work but it's still an option
>>
>> * we still should be able to integrate thru a pipe into Fus, that would
>> make it possible to dump modular and ursine metadata into Fus to perform
>> the dependency solving in a separate subprocess. We should probably
>> re-check the reasons behind our previous decision not to do the same with
>> DNF[2].
>>
>>
>> How is Integration with Fus via pipe (CLI) easier than with gobject?
>> Either way, you "can't directly use the libsolv thingies (pools, solvables
>> and friends)".  Right?  What am I missing?
>>
>>
> Right, a publish-like operation would be required every time, for all
> repositories involved in the copy to dump the metadata to the pipe(s);
> sample of this interface is can be found in Pungi:
> https://pagure.io/pungi/blob/master/f/pungi/wrappers/fus.py the "query"
> is passed thru command line.
> I just learnt Fedora will keep modules and their ursine deps in separate
> repos, so the source repo won't necessarily be closed on dependencies thus
> multiple source repos would be needed.
>
>
> This be done using the Fus gobject interface as well?
>

we'd just dump the XML (and YAML) metadata and run: fus --repo
source1,1,/path/to/pipe1 --repo source2,2,/path/to/pipe2 --repo
target,system,/path/to/target_pipe  "module(walrus)" "penguin:1-2.3" etc
then parse the textual output of fus such as:

# ---%>-
  - nothing provides policycoreutils-python-utils needed by
container-selinux-2:2.69-3.git452b90d.module_2040+0e96cf1b.noarch

Problem 1 / 1:
  - conflicting requests
  - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides libpthread.so.0(GLIBC_2.3.2)(64bit) needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides /bin/bash needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides /usr/bin/python3 needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides python3-dateutil needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
  - nothing provides dbus needed by
atomic-1.22.1-2.module_1637+1872e86a.x86_64
# >%--
(fus:8524): fus-WARNING **: 15:13:09.350: Can't resolve all solvables
module:docker:2017.0:20180816194539:3ff668f0.x86_64@f29
module:container-tools:2017.0:20180816194450:80bd9113.x86_64@f29
*docker-devel-2:1.13.1-61.git9cb56fd.module_2109+7c83ead1.noarch@f29
*containers-common-0.1.31-14.dev.gitb0b750d.module_2040+0e96cf1b.x86_64@f29




>
>>
>> * we should be able to extend current libsolv solver in Pulp,
>> reimplementing the algorithm from Fus. This might be as laborous as the
>> first option. It would probably give us more flexibility as well as more
>> room for screwing things up but the responsibility would be ours alone.
>>
>> Please let me know what option seems more appealing to you; other option
>> suggestion are welcome  too.
>>
>> Cheers,
>> milan
>>
>> [1] https://github.com/fedora-modularity/fus/pull/46
>> [2] https://pulp.plan.io/issues/3528#note-7
>>
>>
>> ___
>> Pulp-dev mailing 
>> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] To integrate Fus or not to....

[Milan Kovacik]

Thanks David!

On Wed, Oct 10, 2018 at 4:59 PM David Davis  wrote:

> One consideration is that whatever we do for Pulp 2, we’ll have to do
> again in Pulp 3. For that reason, as ugly as it is, I am leaning toward
> calling to Fus via a pipe. It seems the easiest, least amount of work, and
> most reusable.
>

Is your concern the amount of work/code the other approaches might require
porting from Pulp2 to Pulp3? I'd say the solver (wrapper) itself should be
Pulp-agnostic up to the feed, the model we use.

Cheers,
milan


>
> David
>
>
> On Wed, Oct 10, 2018 at 10:00 AM Milan Kovacik 
> wrote:
>
>> ...that might be the question we should ask ourselves once again when it
>> comes to recursive copying of units between repositories.
>>
>> I'd like to poll folks opinions about the possibilities that we may have
>> when it comes to integrating third party solvers in Pulp. My yesterday's
>> chat with the #fedora-modularity folks about us integrating the Fus[1]
>> solver in order to reuse the Fus algorithm ran into a couple of bumps:
>>
>> * it would be laborous to create a programmatic Python API between Fus
>> and Pulp because we can't directly use the libsolv thingies (pools,
>> solvables and friends) in such an API because Fus is written utilizing
>> GObject, which is incompatible with Swig, which in turn is used in libsolv
>> to expose the python bindings. One would have to either re-wrap libsolv
>> code in Fus to work with pygobject or submit PRs against libsolv to support
>> GObject introspection. I dunno the details of either approach (yet) but
>> from the sad faces on the IRC and the Fus PR[1] it seemed like a lot of
>> work but it's still an option
>>
>> * we still should be able to integrate thru a pipe into Fus, that would
>> make it possible to dump modular and ursine metadata into Fus to perform
>> the dependency solving in a separate subprocess. We should probably
>> re-check the reasons behind our previous decision not to do the same with
>> DNF[2].
>>
>> * we should be able to extend current libsolv solver in Pulp,
>> reimplementing the algorithm from Fus. This might be as laborous as the
>> first option. It would probably give us more flexibility as well as more
>> room for screwing things up but the responsibility would be ours alone.
>>
>> Please let me know what option seems more appealing to you; other option
>> suggestion are welcome  too.
>>
>> Cheers,
>> milan
>>
>> [1] https://github.com/fedora-modularity/fus/pull/46
>> [2] https://pulp.plan.io/issues/3528#note-7
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] To integrate Fus or not to....

[Milan Kovacik]

...that might be the question we should ask ourselves once again when it
comes to recursive copying of units between repositories.

I'd like to poll folks opinions about the possibilities that we may have
when it comes to integrating third party solvers in Pulp. My yesterday's
chat with the #fedora-modularity folks about us integrating the Fus[1]
solver in order to reuse the Fus algorithm ran into a couple of bumps:

* it would be laborous to create a programmatic Python API between Fus and
Pulp because we can't directly use the libsolv thingies (pools, solvables
and friends) in such an API because Fus is written utilizing GObject, which
is incompatible with Swig, which in turn is used in libsolv to expose the
python bindings. One would have to either re-wrap libsolv code in Fus to
work with pygobject or submit PRs against libsolv to support GObject
introspection. I dunno the details of either approach (yet) but from the
sad faces on the IRC and the Fus PR[1] it seemed like a lot of work but
it's still an option

* we still should be able to integrate thru a pipe into Fus, that would
make it possible to dump modular and ursine metadata into Fus to perform
the dependency solving in a separate subprocess. We should probably
re-check the reasons behind our previous decision not to do the same with
DNF[2].

* we should be able to extend current libsolv solver in Pulp,
reimplementing the algorithm from Fus. This might be as laborous as the
first option. It would probably give us more flexibility as well as more
room for screwing things up but the responsibility would be ours alone.

Please let me know what option seems more appealing to you; other option
suggestion are welcome  too.

Cheers,
milan

[1] https://github.com/fedora-modularity/fus/pull/46
[2] https://pulp.plan.io/issues/3528#note-7
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Recursive copies of modules between repositories

[Milan Kovacik]

Folks,

couple of interesting questions popped up while trying to figure how to
integrate Fus[1] with Pulp to minimize duplicating code, that I'd like to
poll the opinions about:

* how do we infer which platform should the solver consider for evaluating
the modular dependencies, if there are multiple platforms
required/available?

* do we want all the possible alternatives, satisfying a module's modular
dependencies, to be copied or just the first match?

* libsolv-based solvers aren't capable of delivering a best-effort
resolution: a conflict or an unsatisfied dependency will break the
recursive copy, are we cool with that?

Thanks,
milan

[1] https://github.com/fedora-modularity/fus
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

On Fri, Sep 21, 2018 at 4:52 PM Michael Hrivnak  wrote:

> On Thu, Sep 20, 2018 at 9:41 AM, Milan Kovacik 
> wrote:
>
>> Folks,
>>
>> we've been discussing my proposal with Vlada recently and he's about to
>> commit officially to it at the school.
>> I'd like to poke for feedback esp. if there are concerns/doubts about the
>> sanity of my proposal.
>> Encouraging comments are of course welcome too ;)
>>
>
> That could be a very interesting paper. The foundation of the current data
> model and algorithm were created several years ago by myself and Barnaby
> Court. If it would be helpful, I'm happy to talk through how we arrived at
> this solution, what the goals and use cases were, what else we
> considered, etc.
>
>
Mike, that would be invaluable input for the paper, we'd like to get as
much background as possible.


> --
>
> Michael Hrivnak
>
> Principal Software Engineer, RHCE
>
> Red Hat
>

Thanks,
milan
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

On Fri, Sep 21, 2018 at 4:34 PM David Davis  wrote:

> Our previous experiment was to analyze two different ways to store content
> for a repository version: a direct relationship between a version and
> content vs storing diffs between versions (which we have adopted in Pulp
> 3). So we didn’t really examine different branching models and I wouldn’t
> really say we have a branching model today—it’s more of a linear history of
> repository content sets.
>
> I figure it might help if I outline some of my concerns about pursuing a
> branching model.
>
> First is the user experience. Only a small subset of users care about
> being able to branch. If any branching model makes Pulp more difficult to
> use for users who don’t care about branching, that would be a showstopper.
> Having worked with users and other groups at Red Hat, I often worry that
> Pulp is already too difficult to use for some users. I’m much more
> concerned about simplifying Pulp than I am with introducing branching.
>
> Secondly, a branching model would introduce code complexity. Already,
> explaining how repository versions work is probably one of the most
> difficult topics when on-boarding new developers. I don’t see how switching
> to a branching model like git has will make this any easier. Even if we
> mirror git’s model, I am not sure that’s any help as most developers aren’t
> aware of how git works internally.
>
> Another problem is performance. You’ve mentioned this already. Any
> branching model would have to be just as performant as what we have today
> for it to be worth adopting. Perhaps if branching were a central feature to
> Pulp, we would have some more leeway in terms of performance.
>
> Lastly, users can branch today by creating new repositories and copying
> over content. It’s a workaround and not as great as having a branching
> model in the database but Katello uses this method when publishing content
> views. I think it solves most of our users' needs. It’s easy for users to
> understand and it keeps our data model simple.
>
> To kind of sum up things, I can’t imagine a branching model that could
> overcome all of these problems. If the goal is just analyze Pulp's system
> of versioning content vs a model like git while not necessarily committing
> any code to Pulp 3 (except perhaps just some small improvements/tweaks),
> then I’m not opposed to this. I am just doubtful about how productive it
> will be.
>
>
We'd like to get an honest answer to the question: Excuse me, but why not
Git?
We won't presume any conclusions. In case the experiment shows the git-like
approach doesn't add much value, we'll have a solid answer for the question
above.
In any case, examining the details of the current versioning implementation
has value added on its own: just as Robin mentioned above, we could use the
analysis part of the paper to explain current solution to ourselves,
plugin-writers and the users.
Your concerns are valid though and we would like to include them in the
discussion too.

Thanks,
milan


> David
>
>
> On Fri, Sep 21, 2018 at 3:34 AM Milan Kovacik  wrote:
>
>>
>>
>> On Thu, Sep 20, 2018 at 11:14 PM David Davis 
>> wrote:
>>
>>> I’m interested to know more about this versioning/branching proposal and
>>> its goals. I assume this would be to explore changing Pulp 3 to use a
>>> non-linear branching model?
>>>
>>
>> The main goal of the thesis would be to compare Git and Pulp versioning,
>> identifying the advantages and disadvantages of both, when content storage
>> and management is concerned; ideally we'd learn what use cases we are
>> trading for what system traits.
>>
>>
>>> Would the goal be to contribute this model to Pulp 3 codebase in the
>>> near term (like before the MVP)? I guess my concern would be that I don’t
>>> think there is enough of a need nor the time to properly vet it before the
>>> MVP. I don’t think we can guarantee that a branching proposal would result
>>> in a change to Pulp 3 in the near term, and switching the data model
>>> post-MVP is potentially risky and likely backwards incompatible, so
>>> adoption later would be unlikely also.
>>>
>>
>> I think a valuable essay would base the conclusion on an experiment so
>> we'll try to implement (a subset of) the Git data model in Pulp core to get
>> some first-hand experimental data; we'd like to see the code, REST API and
>> performance impact on Pulp core and Pulp plugins.
>> I share your concerns though so we'd work on a fork to avoid pressure on
>> both the experiment and Pulp goals, keeping a focused scope and minimal
>> diff with the project origins; we'd ideally base 

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

On Thu, Sep 20, 2018 at 11:14 PM David Davis  wrote:

> I’m interested to know more about this versioning/branching proposal and
> its goals. I assume this would be to explore changing Pulp 3 to use a
> non-linear branching model?
>

The main goal of the thesis would be to compare Git and Pulp versioning,
identifying the advantages and disadvantages of both, when content storage
and management is concerned; ideally we'd learn what use cases we are
trading for what system traits.


> Would the goal be to contribute this model to Pulp 3 codebase in the near
> term (like before the MVP)? I guess my concern would be that I don’t think
> there is enough of a need nor the time to properly vet it before the MVP. I
> don’t think we can guarantee that a branching proposal would result in a
> change to Pulp 3 in the near term, and switching the data model post-MVP is
> potentially risky and likely backwards incompatible, so adoption later
> would be unlikely also.
>

I think a valuable essay would base the conclusion on an experiment so
we'll try to implement (a subset of) the Git data model in Pulp core to get
some first-hand experimental data; we'd like to see the code, REST API and
performance impact on Pulp core and Pulp plugins.
I share your concerns though so we'd work on a fork to avoid pressure on
both the experiment and Pulp goals, keeping a focused scope and minimal
diff with the project origins; we'd ideally base the fork on an RC (MVP?)
Pulp core tag.
The decision to merge the fork to the origin projects would be based on the
experiment results so it would be out of scope of the thesis.

I also want to point out that we explored another versioning model before
> we went with our current design. We did some performance analysis and found
> that the model wasn’t optimized for the most common use cases in Pulp.
> Here’s the discussion in case you’re interested:
>
> https://www.redhat.com/archives/pulp-dev/2017-December/msg00075.html
>

Great input! I briefly looked at the conversation and your experiment and I
believe we'll utilize this in the comparison indeed.
I'd like to ask though, based on the mailing list discussion, did the
branching experiment setup reflect the most frequent use case i.e a sparse
tree/DAG with few, (long) branches?
Did the experiment consider the Git data model and workflows?


> David
>
>
> On Thu, Sep 20, 2018 at 10:01 AM Robin Chan  wrote:
>
>> Having only this exposure to the bachelor thesis, I've just been reading
>> for my own learning.
>>
>> I'll only share that I would invite Vlada to perhaps write a blog post
>> with some summary type information targetting newcomers with an
>> understanding of git. Such a post might be a way to understand Pulp by
>> allowing them to relate existing understanding to pulp a little quicker.
>>
>> On Thu, Sep 20, 2018 at 9:41 AM, Milan Kovacik 
>> wrote:
>>
>>> Folks,
>>>
>>> we've been discussing my proposal with Vlada recently and he's about to
>>> commit officially to it at the school.
>>> I'd like to poke for feedback esp. if there are concerns/doubts about
>>> the sanity of my proposal.
>>> Encouraging comments are of course welcome too ;)
>>>
>>> Cheers,
>>> milan
>>>
>>> On Mon, Sep 10, 2018 at 11:25 AM Milan Kovacik 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'd like to propose a comparison of Git versioning model to Pulp3
>>>> versioning (and branching) model.
>>>> The textual part would briefly introduce both Git and Pulp, with focus
>>>> on their respective usecases, trying to identify overlapping scenarios.
>>>> Next it would in detail examine the versioning models of both Git and
>>>> Pulp3.
>>>> The main matter would compare implementation of Git versioning model in
>>>> Pulp3 (as e.g a fork) to the current versioning model.
>>>> The textual part would conclude with advantages and disadvantages of
>>>> both versioning models, reiterating over the usecases, comparing
>>>> performance (storage and time) of both the implementations.
>>>>
>>>> I believe this would have a direct practical impact on Pulp3, affecting
>>>> feature planning and if feasible, providing an alternative versioning
>>>> implementation[1][2].
>>>>
>>>> --
>>>> milan
>>>>
>>>> PS: I'm volunteering to oversee the thesis on the project part
>>>>
>>>> [1] https://pulp.plan.io/issues/3360
>>>> [2] https://pulp.plan.io/issues/3842
>>>>
>>>> On Wed, Sep 5, 2018 at 9:16 PM, Austin Macdonald 
>>&

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

On Thu, Sep 20, 2018 at 4:00 PM Robin Chan  wrote:

> Having only this exposure to the bachelor thesis, I've just been reading
> for my own learning.
>
> I'll only share that I would invite Vlada to perhaps write a blog post
> with some summary type information targetting newcomers with an
> understanding of git. Such a post might be a way to understand Pulp by
> allowing them to relate existing understanding to pulp a little quicker.
>

great idea, a blog post or even a presentation comparing the details and
trades of both the systems could be a nice introduction to Pulp versioning,
valuable for both to the users and the plugin writers.


> On Thu, Sep 20, 2018 at 9:41 AM, Milan Kovacik 
> wrote:
>
>> Folks,
>>
>> we've been discussing my proposal with Vlada recently and he's about to
>> commit officially to it at the school.
>> I'd like to poke for feedback esp. if there are concerns/doubts about the
>> sanity of my proposal.
>> Encouraging comments are of course welcome too ;)
>>
>> Cheers,
>> milan
>>
>> On Mon, Sep 10, 2018 at 11:25 AM Milan Kovacik 
>> wrote:
>>
>>> Hi,
>>>
>>> I'd like to propose a comparison of Git versioning model to Pulp3
>>> versioning (and branching) model.
>>> The textual part would briefly introduce both Git and Pulp, with focus
>>> on their respective usecases, trying to identify overlapping scenarios.
>>> Next it would in detail examine the versioning models of both Git and
>>> Pulp3.
>>> The main matter would compare implementation of Git versioning model in
>>> Pulp3 (as e.g a fork) to the current versioning model.
>>> The textual part would conclude with advantages and disadvantages of
>>> both versioning models, reiterating over the usecases, comparing
>>> performance (storage and time) of both the implementations.
>>>
>>> I believe this would have a direct practical impact on Pulp3, affecting
>>> feature planning and if feasible, providing an alternative versioning
>>> implementation[1][2].
>>>
>>> --
>>> milan
>>>
>>> PS: I'm volunteering to oversee the thesis on the project part
>>>
>>> [1] https://pulp.plan.io/issues/3360
>>> [2] https://pulp.plan.io/issues/3842
>>>
>>> On Wed, Sep 5, 2018 at 9:16 PM, Austin Macdonald 
>>> wrote:
>>>
>>>> How about reproducibility? Bihan and I presented on this topic at scipy
>>>> but there is a lot left to say/research.
>>>> https://www.youtube.com/watch?v=5czGgUG0oXA
>>>>
>>>> In the scientific community, reproducibility is a hot topic (I bet grad
>>>> schools would love this!), and the computational aspect of it has been
>>>> under-emphasized in the existing literature. Our talk focused on the
>>>> fundamental problems of environmental reproducibility and how to use Pulp
>>>> to solve them. Especially since the scope was narrow, we barely scratched
>>>> the surface. There a ton of potential angles; some ideas are that you could
>>>> catalog and compare technologies or discuss academic vs industry use cases
>>>> and trends.
>>>>
>>>> On Wed, Sep 5, 2018 at 5:01 AM Vladimir Dusek 
>>>> wrote:
>>>>
>>>>> I don't ask you for a specific assignment, we'll set up it with Milan
>>>>> later. I only need some idea on a topic.
>>>>>
>>>>> Scope of the work should be 40-50 standard pages. Additional
>>>>> istructions could be find here -
>>>>> http://www.fit.vutbr.cz/info/szz/pokyny_bp.php.en#pozadavky. However
>>>>> I don't think it's neccessary for you to know them.
>>>>>
>>>>> All bachelor theses from our university last year could be find here -
>>>>> http://www.fit.vutbr.cz/study/DP/BP.php.en. I presume most of them
>>>>> are written in Czech but there are abstracts in English which might be
>>>>> useful for inspiration.
>>>>>
>>>>> Random abstracts:
>>>>>
>>>>> Alias Analysis in C Compiler
>>>>> -
>>>>> This thesis is dedicated to the problem of alias analysis and
>>>>> possibilities of its improvement in the LLVM framework. The goal of this
>>>>> thesis is to improve the accuracy, which was achieved by extending the
>>>>> existing implementation of Andresen algorithm to be field sensitive. The
>>>>> terms related

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

On Thu, Sep 20, 2018 at 3:53 PM Dana Walker  wrote:

> This sounds interesting, and a good way to explore the branching model
> suggestions in the referenced tickets
>
yes, that's one of the goals indeed



> I look forward to reading your thesis Vlada!
>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> <https://www.redhat.com>
> <https://red.ht/sig>
>
> On Thu, Sep 20, 2018 at 9:41 AM, Milan Kovacik 
> wrote:
>
>> Folks,
>>
>> we've been discussing my proposal with Vlada recently and he's about to
>> commit officially to it at the school.
>> I'd like to poke for feedback esp. if there are concerns/doubts about the
>> sanity of my proposal.
>> Encouraging comments are of course welcome too ;)
>>
>> Cheers,
>> milan
>>
>> On Mon, Sep 10, 2018 at 11:25 AM Milan Kovacik 
>> wrote:
>>
>>> Hi,
>>>
>>> I'd like to propose a comparison of Git versioning model to Pulp3
>>> versioning (and branching) model.
>>> The textual part would briefly introduce both Git and Pulp, with focus
>>> on their respective usecases, trying to identify overlapping scenarios.
>>> Next it would in detail examine the versioning models of both Git and
>>> Pulp3.
>>> The main matter would compare implementation of Git versioning model in
>>> Pulp3 (as e.g a fork) to the current versioning model.
>>> The textual part would conclude with advantages and disadvantages of
>>> both versioning models, reiterating over the usecases, comparing
>>> performance (storage and time) of both the implementations.
>>>
>>> I believe this would have a direct practical impact on Pulp3, affecting
>>> feature planning and if feasible, providing an alternative versioning
>>> implementation[1][2].
>>>
>>> --
>>> milan
>>>
>>> PS: I'm volunteering to oversee the thesis on the project part
>>>
>>> [1] https://pulp.plan.io/issues/3360
>>> [2] https://pulp.plan.io/issues/3842
>>>
>>> On Wed, Sep 5, 2018 at 9:16 PM, Austin Macdonald 
>>> wrote:
>>>
>>>> How about reproducibility? Bihan and I presented on this topic at scipy
>>>> but there is a lot left to say/research.
>>>> https://www.youtube.com/watch?v=5czGgUG0oXA
>>>>
>>>> In the scientific community, reproducibility is a hot topic (I bet grad
>>>> schools would love this!), and the computational aspect of it has been
>>>> under-emphasized in the existing literature. Our talk focused on the
>>>> fundamental problems of environmental reproducibility and how to use Pulp
>>>> to solve them. Especially since the scope was narrow, we barely scratched
>>>> the surface. There a ton of potential angles; some ideas are that you could
>>>> catalog and compare technologies or discuss academic vs industry use cases
>>>> and trends.
>>>>
>>>> On Wed, Sep 5, 2018 at 5:01 AM Vladimir Dusek 
>>>> wrote:
>>>>
>>>>> I don't ask you for a specific assignment, we'll set up it with Milan
>>>>> later. I only need some idea on a topic.
>>>>>
>>>>> Scope of the work should be 40-50 standard pages. Additional
>>>>> istructions could be find here -
>>>>> http://www.fit.vutbr.cz/info/szz/pokyny_bp.php.en#pozadavky. However
>>>>> I don't think it's neccessary for you to know them.
>>>>>
>>>>> All bachelor theses from our university last year could be find here -
>>>>> http://www.fit.vutbr.cz/study/DP/BP.php.en. I presume most of them
>>>>> are written in Czech but there are abstracts in English which might be
>>>>> useful for inspiration.
>>>>>
>>>>> Random abstracts:
>>>>>
>>>>> Alias Analysis in C Compiler
>>>>> -
>>>>> This thesis is dedicated to the problem of alias analysis and
>>>>> possibilities of its improvement in the LLVM framework. The goal of this
>>>>> thesis is to improve the accuracy, which was achieved by extending the
>>>>> existing implementation of Andresen algorithm to be field sensitive. The
>>>>> terms related to alias analysis and algorithms of the alias analysis
>>>>> available in LLVM are explained. These algorithms are compared according 
>>>>> to
>>>>> their base idea, featu

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

Folks,

we've been discussing my proposal with Vlada recently and he's about to
commit officially to it at the school.
I'd like to poke for feedback esp. if there are concerns/doubts about the
sanity of my proposal.
Encouraging comments are of course welcome too ;)

Cheers,
milan

On Mon, Sep 10, 2018 at 11:25 AM Milan Kovacik  wrote:

> Hi,
>
> I'd like to propose a comparison of Git versioning model to Pulp3
> versioning (and branching) model.
> The textual part would briefly introduce both Git and Pulp, with focus on
> their respective usecases, trying to identify overlapping scenarios.
> Next it would in detail examine the versioning models of both Git and
> Pulp3.
> The main matter would compare implementation of Git versioning model in
> Pulp3 (as e.g a fork) to the current versioning model.
> The textual part would conclude with advantages and disadvantages of both
> versioning models, reiterating over the usecases, comparing performance
> (storage and time) of both the implementations.
>
> I believe this would have a direct practical impact on Pulp3, affecting
> feature planning and if feasible, providing an alternative versioning
> implementation[1][2].
>
> --
> milan
>
> PS: I'm volunteering to oversee the thesis on the project part
>
> [1] https://pulp.plan.io/issues/3360
> [2] https://pulp.plan.io/issues/3842
>
> On Wed, Sep 5, 2018 at 9:16 PM, Austin Macdonald 
> wrote:
>
>> How about reproducibility? Bihan and I presented on this topic at scipy
>> but there is a lot left to say/research.
>> https://www.youtube.com/watch?v=5czGgUG0oXA
>>
>> In the scientific community, reproducibility is a hot topic (I bet grad
>> schools would love this!), and the computational aspect of it has been
>> under-emphasized in the existing literature. Our talk focused on the
>> fundamental problems of environmental reproducibility and how to use Pulp
>> to solve them. Especially since the scope was narrow, we barely scratched
>> the surface. There a ton of potential angles; some ideas are that you could
>> catalog and compare technologies or discuss academic vs industry use cases
>> and trends.
>>
>> On Wed, Sep 5, 2018 at 5:01 AM Vladimir Dusek  wrote:
>>
>>> I don't ask you for a specific assignment, we'll set up it with Milan
>>> later. I only need some idea on a topic.
>>>
>>> Scope of the work should be 40-50 standard pages. Additional istructions
>>> could be find here -
>>> http://www.fit.vutbr.cz/info/szz/pokyny_bp.php.en#pozadavky. However I
>>> don't think it's neccessary for you to know them.
>>>
>>> All bachelor theses from our university last year could be find here -
>>> http://www.fit.vutbr.cz/study/DP/BP.php.en. I presume most of them are
>>> written in Czech but there are abstracts in English which might be useful
>>> for inspiration.
>>>
>>> Random abstracts:
>>>
>>> Alias Analysis in C Compiler
>>> -
>>> This thesis is dedicated to the problem of alias analysis and
>>> possibilities of its improvement in the LLVM framework. The goal of this
>>> thesis is to improve the accuracy, which was achieved by extending the
>>> existing implementation of Andresen algorithm to be field sensitive. The
>>> terms related to alias analysis and algorithms of the alias analysis
>>> available in LLVM are explained. These algorithms are compared according to
>>> their base idea, features, and limitations. The implementation of the field
>>> sensitivity has been tested using compiler test suites. Its impact on
>>> program compilation speed and performance has been analyzed. The measured
>>> results show an increase in the accuracy of alias analysis in the LLVM
>>> framework.
>>>
>>> Reinforcement Learning for Starcraft Game Playing
>>> 
>>> This work focuses on methods of machine learning for playing real-time
>>> strategy games. The thesis applies mainly methods of Q-learning based on
>>> reinforcement learning. The practical part of this work is implementing an
>>> agent for playing Starcraft II. Mine solution is based on 4 simple
>>> networks, that are colaborating together. Each of the network also teaches
>>> itself how to process all given actions optimally. Analysis of the system
>>> is based on experiments and statistics from played games.
>>>
>>>
>>> On Tue, Sep 4, 2018 at 8:27 PM Robin Chan  wrote:
>>>
>>>> Hi Vladimir,
>>>>
>>&

Re: [Pulp-dev] commit-bit nomination

[Milan Kovacik]

t;>>>> A & B can be reduced through the passage of time.  Having community
>>>>>> members who are active and in it for the long haul, who have demonstrated
>>>>>> good practices and gained experience such that you feel they are less
>>>>>> likely to make the mistakes of B or to be the malicious actor of A.
>>>>>>
>>>>>>
>>>>>> --What does this mean for the commit bit process?
>>>>>>
>>>>>> What we can't mitigate in other ways and really are trying to control
>>>>>> is time and commitment.  We want to see people who care about the project
>>>>>> more than just today for one contribution, who are comfortable with the
>>>>>> process, and would add to the positive collaboration going on.  This is
>>>>>> outlined in our PUP, and one of the reasons we left this subjective was 
>>>>>> so
>>>>>> that folks wouldn't try to game a system if it was automated based on
>>>>>> numbers.   For some people, this process may take one month, for others 
>>>>>> one
>>>>>> year, but anyone who meets these qualifications should be given the 
>>>>>> commit
>>>>>> bit so that we can continue to grow.
>>>>>>
>>>>>> However, this subjective assessment shouldn't be a really high bar.
>>>>>> We're not seeking epic level contributions, we're not here to limit this
>>>>>> project to only principle devs.  Even someone starting out may start with
>>>>>> an open source project, and as they contribute more and more, with filing
>>>>>> bugs, submitting pull requests, engaging in discussions, and providing
>>>>>> constructive feedback, they too should be qualified to review and approve
>>>>>> code without having to make a full-time job of it, or we might as well go
>>>>>> back to having employees have the commit bit only.
>>>>>>
>>>>>> This project is too large to limit commit bits/code reviews/merging
>>>>>> only to individuals who know the workings of the entire repository inside
>>>>>> and out.  Is there a chance they could miss something in a review that 
>>>>>> has
>>>>>> an unanticipated negative impact elsewhere in the code?  Sure, as with
>>>>>> anyone, but that's also what tests, QE, teammates, and bugfixes are for.
>>>>>> We need trust, not control beyond the basic level to mitigate risks.
>>>>>>
>>>>>> 
>>>>>>
>>>>>>
>>>>>> Either way, this type of conversation is probably best suited for
>>>>>>> discussion amongst each plugin group as they determine what their needs 
>>>>>>> are.
>>>>>>>
>>>>>>
>>>>>> Here I agree with you.  I had assumed if you were qualified to
>>>>>> receive the commit bit for Pulp, you'd be qualified to have it in other
>>>>>> areas, especially as I've seen folks from the mini teams requesting
>>>>>> cross-team help when they're overloaded, but if we really want plugins to
>>>>>> have autonomy, then this vote should really be focused on pulp then the
>>>>>> pulp_rpm team can make their own call in a separate procedure.
>>>>>>
>>>>>>
>>>>>> As for this nomination specifically, @milan has demonstrated each of
>>>>>> the points to becoming a committer listed here. [0]  Therefore, I am 
>>>>>> still
>>>>>> +1 on giving him the commit bit.*
>>>>>>
>>>>>>
>>>>>>
>>>>>> [0] https://github.com/pulp/pups/blob/master/pup-0006.md
>>>>>> * I do not have the commit bit yet myself, so technically my vote
>>>>>> does not count, but I think this was a very important discussion to have
>>>>>> given the new process and the potential effects these decisions will have
>>>>>> on the health of the Pulp project community, and thus eventually on my 
>>>>>> job.
>>>>>>
>>>>>>
>>>>>> Respectfully,
>>>>>>
>>>>>> --Dana
>>>>>>
>>>>>>
>>>>>> Dana Walker
>>>>>>
>>>>>> Associate Software Engineer
>>>>>>
>>>>>> Red Hat
>>>>>>
>>>>>> <https://www.redhat.com>
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>> On Thu, Sep 6, 2018 at 8:26 AM, Brian Bouterse 
>>>>>> wrote:
>>>>>>
>>>>>>> I was hoping for more discussion before the calling of a vote. I'm a
>>>>>>> committer in both of these areas, and this nomination surprised me since
>>>>>>> we've never talked about either change. I have concerns both with
>>>>>>> increasing the sizes of these teams and also with this specific 
>>>>>>> nomination
>>>>>>> at this time.
>>>>>>>
>>>>>>> With Pulp2 nearing maintenance mode, the core and plugin teams need
>>>>>>> to assess what their needs are both in code and people. I feel that 
>>>>>>> with 9
>>>>>>> people on the core team, maintaining vision is hard/impossible, and the
>>>>>>> mailing list discussions have demonstrated that. Also consider that 
>>>>>>> Pulp3
>>>>>>> is an order of magnitude smaller codebase (literally) so keeping the 
>>>>>>> same
>>>>>>> number of committers seems too many. I also feel that any committer 
>>>>>>> being
>>>>>>> added should already be very involved, contributing features and
>>>>>>> discussion, not added and then gotten involved. Milan has been very
>>>>>>> involved in Pulp2 RPM work, but Pulp2 and Pulp3 are very different.
>>>>>>> pulp_rpm for Pulp3 is nearly feature complete and Milan hasn't been
>>>>>>> involved in any of that planning or implementation. Maybe the Pulp3/2 
>>>>>>> teams
>>>>>>> should be split? Either way, this type of conversation is probably best
>>>>>>> suited for discussion amongst each plugin group as they determine what
>>>>>>> their needs are.
>>>>>>>
>>>>>>> @Milan I think one area the core team does need leadership now is on
>>>>>>> pulp2->3 migration planning. I think that would be a good way to get 
>>>>>>> more
>>>>>>> involved as a non-committer to demonstrate epic-level feature planning,
>>>>>>> show more collaboration, and clearer communication. Those are the main
>>>>>>> aspects I'm looking for. I can give feedback directly too if that's 
>>>>>>> helpful.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 5, 2018 at 10:44 AM, Daniel Alley 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Now that PUP 6 has been merged, it's time to follow the process :)
>>>>>>>>
>>>>>>>> I would like to nominate Milan Kovacik (mkova...@redhat.com) for a
>>>>>>>> commit bit on these repositories:
>>>>>>>>
>>>>>>>>
>>>>>>>>- pulp
>>>>>>>>- pulp_rpm
>>>>>>>>- devel
>>>>>>>>
>>>>>>>>
>>>>>>>> Milan has demonstrated a consistent dedication to quality in his
>>>>>>>> contributions and his reviews.  I believe his work speaks for itself 
>>>>>>>> :).
>>>>>>>>
>>>>>>>> The vote end date is September 12th, seven days from today.
>>>>>>>>
>>>>>>>> ___
>>>>>>>> Pulp-dev mailing list
>>>>>>>> Pulp-dev@redhat.com
>>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ___
>>>>>>> Pulp-dev mailing list
>>>>>>> Pulp-dev@redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ___
>>>>>> Pulp-dev mailing list
>>>>>> Pulp-dev@redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>>
>>>>>>
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Theme of the bachelor thesis

[Milan Kovacik]

Hi,

I'd like to propose a comparison of Git versioning model to Pulp3
versioning (and branching) model.
The textual part would briefly introduce both Git and Pulp, with focus on
their respective usecases, trying to identify overlapping scenarios.
Next it would in detail examine the versioning models of both Git and Pulp3.
The main matter would compare implementation of Git versioning model in
Pulp3 (as e.g a fork) to the current versioning model.
The textual part would conclude with advantages and disadvantages of both
versioning models, reiterating over the usecases, comparing performance
(storage and time) of both the implementations.

I believe this would have a direct practical impact on Pulp3, affecting
feature planning and if feasible, providing an alternative versioning
implementation[1][2].

--
milan

PS: I'm volunteering to oversee the thesis on the project part

[1] https://pulp.plan.io/issues/3360
[2] https://pulp.plan.io/issues/3842

On Wed, Sep 5, 2018 at 9:16 PM, Austin Macdonald  wrote:

> How about reproducibility? Bihan and I presented on this topic at scipy
> but there is a lot left to say/research. https://www.youtube.com/watch?
> v=5czGgUG0oXA
>
> In the scientific community, reproducibility is a hot topic (I bet grad
> schools would love this!), and the computational aspect of it has been
> under-emphasized in the existing literature. Our talk focused on the
> fundamental problems of environmental reproducibility and how to use Pulp
> to solve them. Especially since the scope was narrow, we barely scratched
> the surface. There a ton of potential angles; some ideas are that you could
> catalog and compare technologies or discuss academic vs industry use cases
> and trends.
>
> On Wed, Sep 5, 2018 at 5:01 AM Vladimir Dusek  wrote:
>
>> I don't ask you for a specific assignment, we'll set up it with Milan
>> later. I only need some idea on a topic.
>>
>> Scope of the work should be 40-50 standard pages. Additional istructions
>> could be find here - http://www.fit.vutbr.cz/info/
>> szz/pokyny_bp.php.en#pozadavky. However I don't think it's neccessary
>> for you to know them.
>>
>> All bachelor theses from our university last year could be find here -
>> http://www.fit.vutbr.cz/study/DP/BP.php.en. I presume most of them are
>> written in Czech but there are abstracts in English which might be useful
>> for inspiration.
>>
>> Random abstracts:
>>
>> Alias Analysis in C Compiler
>> -
>> This thesis is dedicated to the problem of alias analysis and
>> possibilities of its improvement in the LLVM framework. The goal of this
>> thesis is to improve the accuracy, which was achieved by extending the
>> existing implementation of Andresen algorithm to be field sensitive. The
>> terms related to alias analysis and algorithms of the alias analysis
>> available in LLVM are explained. These algorithms are compared according to
>> their base idea, features, and limitations. The implementation of the field
>> sensitivity has been tested using compiler test suites. Its impact on
>> program compilation speed and performance has been analyzed. The measured
>> results show an increase in the accuracy of alias analysis in the LLVM
>> framework.
>>
>> Reinforcement Learning for Starcraft Game Playing
>> 
>> This work focuses on methods of machine learning for playing real-time
>> strategy games. The thesis applies mainly methods of Q-learning based on
>> reinforcement learning. The practical part of this work is implementing an
>> agent for playing Starcraft II. Mine solution is based on 4 simple
>> networks, that are colaborating together. Each of the network also teaches
>> itself how to process all given actions optimally. Analysis of the system
>> is based on experiments and statistics from played games.
>>
>>
>> On Tue, Sep 4, 2018 at 8:27 PM Robin Chan  wrote:
>>
>>> Hi Vladimir,
>>>
>>> I'm not familiar with what types of topics are appropriate for this type
>>> of project - could you share some criteria and examples of what makes a
>>> good topic?
>>>
>>> Robin
>>>
>>> On Mon, Sep 3, 2018 at 8:50 AM, Vladimir Dusek 
>>> wrote:
>>>
 Hi team,

 since September I'm in the third grade of bachelor studies and it means
 that I'm going to write a bachelor thesis. I haven't selected a topic yet
 and if it'll be possible I'd like to work on something that is relevant to
 Pulp. Do you have any idea?

 Thank you guys,
 Vlada

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> 

Re: [Pulp-dev] commit-bit nomination

[Milan Kovacik]

Dana, Daniel,

thanks guys, I really appreciate the recognition! :)

Cheers,
milan

PS: my GitHub https://github.com/dparalen

On Wed, Sep 5, 2018 at 4:56 PM, Dana Walker  wrote:

> +1
>
> Enthusiastically!
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> <https://www.redhat.com>
> <https://red.ht/sig>
>
> On Wed, Sep 5, 2018 at 10:44 AM, Daniel Alley  wrote:
>
>> Now that PUP 6 has been merged, it's time to follow the process :)
>>
>> I would like to nominate Milan Kovacik (mkova...@redhat.com) for a
>> commit bit on these repositories:
>>
>>
>>- pulp
>>- pulp_rpm
>>- devel
>>
>>
>> Milan has demonstrated a consistent dedication to quality in his
>> contributions and his reviews.  I believe his work speaks for itself :).
>>
>> The vote end date is September 12th, seven days from today.
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Commit bit PUP-6

[Milan Kovacik]

+1

Thanks Ina!
milan

On Fri, Aug 24, 2018 at 8:44 PM, Brian Bouterse  wrote:

> I left some more feedback. The only substantive request is
> s/supermajority/two-thirds/ since supermajority can mean several things
> per:  https://en.wikipedia.org/wiki/Supermajority
>
> Assuming those comments are addressed, I'm +1.
>
> Thank you so much for putting this together @ipanova!
>
>
> On Fri, Aug 24, 2018 at 9:59 AM, Ina Panova  wrote:
>
>> I tried to address all the comments and updates the PR.
>>
>> Please give another look and vote once you're ready!
>>
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>> On Thu, Aug 23, 2018 at 5:54 PM, Dana Walker  wrote:
>>
>>> +1
>>>
>>> I'm a little concerned about the stated drawbacks, but even the
>>> alternatives can have issues, so I think this is a good place to start and
>>> we can amend the process later if we run into problems and/or we arrive at
>>> a good solution to mitigate the drawbacks.
>>>
>>> --Dana
>>>
>>> Dana Walker
>>>
>>> Associate Software Engineer
>>>
>>> Red Hat
>>>
>>> 
>>> 
>>>
>>> On Fri, Aug 17, 2018 at 4:13 PM, Robin Chan  wrote:
>>>
 +1
 I like the clarification to the removal piece. You may accept the
 suggestions if they are helpful.




 On Fri, Aug 17, 2018 at 9:32 AM, David Davis 
 wrote:

> Assuming there are no major changes to the PUP, I’ll vote +1.
>
> Thanks for putting this together.
>
> David
>
>
> On Fri, Aug 17, 2018 at 7:21 AM Ina Panova  wrote:
>
>> PR updated,
>> ready for re-review.
>>
>>
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>> On Thu, Aug 16, 2018 at 1:00 PM, Ina Panova 
>> wrote:
>>
>>> I have opened a PR for PUP-6 [0]  which describes the commit bit
>>> assignment/revocation process.
>>>
>>> Please take a look to review and provide feedback.
>>>
>>> I'd like to call for a vote by August 28, 2018. Per PUP-1[1], are
>>> the voting options:
>>>
>>> +1: "Will benefit the project and should definitely be adopted."
>>> +0: "Might benefit the project and is acceptable."
>>> -0: "Might not be the right choice but is acceptable."
>>> -1: "I have serious reservations that need to be thought through and
>>> addressed."
>>>
>>> Thank you!
>>>
>>> [0] https://github.com/pulp/pups/pull/15
>>> [1] https://github.com/pulp/pups/blob/master/pup-0001.md
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 5:11 PM, Dana Walker  wrote:

> Can this idea of SME as code owner be used *only* for notification so that
> we don't get flooded with them for every PR across the codebase and *not*
> as a requirement for merging so other still can get things rolling, this
> just covers the scenarios where a PR fell between the cracks unnoticed?
>

lazy myself excuse would be: meh I'm not an SME here, I don't review that
PR part

My concern is we should use the tools in a way that motivate reviews to
both spread the domain knowledge (i.e to become a commit bit owner and/or
an SME eventually) and help the review process speed and quality.

I suggest adopting formal criteria that would promote this; implementation
can be thru Code owners just mind to create/update the codeowners file a
commit bit is required too.

--
milan


>
> --Dana
>
> Dana Walker
>
> Associate Software Engineer
>
> Red Hat
>
> 
> 
>
> On Tue, Aug 14, 2018 at 11:05 AM, David Davis 
> wrote:
>
>> The code owner feature in Github requests reviews from code owners when a
>> PR is opened. This both notifies the responsible team when they have a PR
>> to review and also makes it clear who should be reviewing a particular PR
>> so that the PR author can follow up with those people.
>>
>> David
>>
>>
>> On Tue, Aug 14, 2018 at 10:36 AM Bryan Kearney 
>> wrote:
>>
>>> On 08/13/2018 05:29 PM, David Davis wrote:
>>> >
>>> >
>>> > # Problem Statement
>>> >
>>> > For Pulp's review process, there are several areas we could improve:
>>> >
>>> > 1. It’s not always clear who should review files. Over time we have
>>> > developed subject matter experts for different areas of the codebase,
>>> > but these are not codified anywhere. It would be useful for us to
>>> define
>>> > teams need to review projects using code owners.
>>> >
>>> > 2. PRs go unnoticed. Github has a request-review feature, but only
>>> > members of the github organization can click 'request review' button.
>>> It
>>> > would be great if when a PR is opened people automatically received PR
>>> > review requests.
>>> >
>>> >
>>> > # Team Examples
>>> >
>>> > Functional Tests - The QE Team should be code owners of functional
>>> tests
>>> > that test core or core-maintained plugins
>>> > The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
>>> > this area
>>> >
>>> >
>>> > # Solution
>>> >
>>> > 1. Configure the code-owners feature of Github
>>> > (https://blog.github.com/2017-07-06-introducing-code-owners/). It will
>>> > allow a team of 2 or more people to be notified and asked for review
>>> > when a PR modifies a file within a certain area of the code.
>>> >
>>> > 2. Require code-owner review to merge. This is described in this
>>> > section:
>>> > https://blog.github.com/2017-07-06-introducing-code-owners/#
>>> an-extra-layer-of-code-security
>>> >
>>> >
>>> > # Process
>>> >
>>> > The code owner role is not related to the commit bit. It's designed to
>>> > get better reviews. Well reviewed work can be confidently merged by
>>> > anyone with the commit bit.
>>> >
>>> > To make a change to code owners, open a PR with the changes, and call
>>> > for a lazy consensus vote by mailing list. Similar to the PUP decision
>>> > making process, voting must be open for 10 days, and the committers of
>>> > the respective repository are voting.
>>> >
>>> > The code owners file itself should be owned by the core committers of
>>> > the repository.
>>> >
>>> If the problem statement is a slowdown in PRs, how does limiting who can
>>> do the review/merge solve the issue?
>>>
>>> -- bk
>>>
>>>
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 5:05 PM, David Davis  wrote:

> The code owner feature in Github requests reviews from code owners when a
> PR is opened. This both notifies the responsible team when they have a PR
> to review and also makes it clear who should be reviewing a particular PR
> so that the PR author can follow up with those people.
>

for this to work there has to be some initial set of SMEs to be put in the
code ownership file, correct?
Is it going to be the commit bit owners by default?



>
> David
>
>
> On Tue, Aug 14, 2018 at 10:36 AM Bryan Kearney 
> wrote:
>
>> On 08/13/2018 05:29 PM, David Davis wrote:
>> >
>> >
>> > # Problem Statement
>> >
>> > For Pulp's review process, there are several areas we could improve:
>> >
>> > 1. It’s not always clear who should review files. Over time we have
>> > developed subject matter experts for different areas of the codebase,
>> > but these are not codified anywhere. It would be useful for us to define
>> > teams need to review projects using code owners.
>> >
>> > 2. PRs go unnoticed. Github has a request-review feature, but only
>> > members of the github organization can click 'request review' button. It
>> > would be great if when a PR is opened people automatically received PR
>> > review requests.
>> >
>> >
>> > # Team Examples
>> >
>> > Functional Tests - The QE Team should be code owners of functional tests
>> > that test core or core-maintained plugins
>> > The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
>> > this area
>> >
>> >
>> > # Solution
>> >
>> > 1. Configure the code-owners feature of Github
>> > (https://blog.github.com/2017-07-06-introducing-code-owners/). It will
>> > allow a team of 2 or more people to be notified and asked for review
>> > when a PR modifies a file within a certain area of the code.
>> >
>> > 2. Require code-owner review to merge. This is described in this
>> > section:
>> > https://blog.github.com/2017-07-06-introducing-code-owners/
>> #an-extra-layer-of-code-security
>> >
>> >
>> > # Process
>> >
>> > The code owner role is not related to the commit bit. It's designed to
>> > get better reviews. Well reviewed work can be confidently merged by
>> > anyone with the commit bit.
>> >
>> > To make a change to code owners, open a PR with the changes, and call
>> > for a lazy consensus vote by mailing list. Similar to the PUP decision
>> > making process, voting must be open for 10 days, and the committers of
>> > the respective repository are voting.
>> >
>> > The code owners file itself should be owned by the core committers of
>> > the repository.
>> >
>> If the problem statement is a slowdown in PRs, how does limiting who can
>> do the review/merge solve the issue?
>>
>> -- bk
>>
>>
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 4:36 PM, Bryan Kearney  wrote:

> On 08/13/2018 05:29 PM, David Davis wrote:
> >
> >
> > # Problem Statement
> >
> > For Pulp's review process, there are several areas we could improve:
> >
> > 1. It’s not always clear who should review files. Over time we have
> > developed subject matter experts for different areas of the codebase,
> > but these are not codified anywhere. It would be useful for us to define
> > teams need to review projects using code owners.
> >
> > 2. PRs go unnoticed. Github has a request-review feature, but only
> > members of the github organization can click 'request review' button. It
> > would be great if when a PR is opened people automatically received PR
> > review requests.
> >
> >
> > # Team Examples
> >
> > Functional Tests - The QE Team should be code owners of functional tests
> > that test core or core-maintained plugins
> > The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
> > this area
> >
> >
> > # Solution
> >
> > 1. Configure the code-owners feature of Github
> > (https://blog.github.com/2017-07-06-introducing-code-owners/). It will
> > allow a team of 2 or more people to be notified and asked for review
> > when a PR modifies a file within a certain area of the code.
> >
> > 2. Require code-owner review to merge. This is described in this
> > section:
> > https://blog.github.com/2017-07-06-introducing-code-owners/
> #an-extra-layer-of-code-security
> >
> >
> > # Process
> >
> > The code owner role is not related to the commit bit. It's designed to
> > get better reviews. Well reviewed work can be confidently merged by
> > anyone with the commit bit.
> >
> > To make a change to code owners, open a PR with the changes, and call
> > for a lazy consensus vote by mailing list. Similar to the PUP decision
> > making process, voting must be open for 10 days, and the committers of
> > the respective repository are voting.
> >
> > The code owners file itself should be owned by the core committers of
> > the repository.
> >
> If the problem statement is a slowdown in PRs, how does limiting who can
> do the review/merge solve the issue?
>
> hard unless the SME (and/or commit bit ownership) status is tied formally
to review quality and rate.


-- bk
>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 4:29 PM, Ina Panova  wrote:

> +1 for the pup.
>
> Milan,
>
> I guess you are SME, when you are publicly recognized to understand the
> topic.
> You will ask *when* this public recognition is happening?  When the answer
> to the such question like:
> -Who is person to contact for X?
> And the answer will be :
> - It's this guy Y.
>

How does one become the guy Y?
Why isn't guy Y a commit bit owner, if he/she's essential for the
commit/non-commit decision process?

--
milan


>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Tue, Aug 14, 2018 at 3:56 PM, Milan Kovacik 
> wrote:
>
>>
>>
>> On Tue, Aug 14, 2018 at 3:47 PM, David Davis 
>> wrote:
>>
>>> On Tue, Aug 14, 2018 at 9:35 AM Milan Kovacik 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 14, 2018 at 1:26 PM, David Davis 
>>>> wrote:
>>>>
>>>>> The relevant party could either be a subset of the commit bit owners
>>>>> (e.g. task group) or a set of people who don’t have the commit bit (e.g.
>>>>> QE). See the team examples from my original email.
>>>>>
>>>>
>>>>  So what you mean is actually a trusted subset of commit bit owners,
>>>> like the SMEs?
>>>>
>>>
>>> These teams aren’t necessarily a subset of commit bit owners but yes
>>> they’ll be subject matter experts (SMEs) for the code they own. Take QE for
>>> example. They might not have the commit bit to the pulp repo but they are
>>> still the SMEs for pulp_smash tests and thus they’ll probably be code
>>> owners for the smash tests in pulp and pulp_file.
>>>
>>>
>>>>  So we don't trust all commit bit owners equally when it comes to
>>>> particular git (sub)tree?
>>>>
>>>  And we trust (by blocking the merge) on e.g QE approving a PR more than
>>>> the commit bit owner that is outside of the subset?
>>>>  Or is it rather about decoupling the code review duty from the commit
>>>> bit ownership?
>>>>
>>>
>>> To answer these questions, I don’t think it’s about trust. It’s about
>>> (as you mention) decoupling merging code from code reviews. We want to make
>>> sure the appropriate people get notified and have a chance to review the
>>> PRs for which they are SMEs.
>>>
>>
>> This has the same issue as the commit bit ownership, it's just more fine
>> grained and bound to particular subtrees: how does one become an SME?
>> What's the SME lifecycle?
>>
>>
>>>
>>>
>>>>  Why do we need commit bit owners then?
>>>>
>>>
>>> How else do we merge the code if no one has a commit bit?
>>>
>>
>>   thru a bot for instance
>>
>>
>>>
>>>
>>>>
>>>> Cheers,
>>>> milan
>>>>
>>>>
>>>>>
>>>>> Daniel, you are correct. The only caveat is that PRs can’t be merged
>>>>> if they touch a file owned by a team and haven’t been approved by that 
>>>>> team.
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>> On Tue, Aug 14, 2018 at 6:35 AM Milan Kovacik 
>>>>> wrote:
>>>>>
>>>>>> +0 who's the relevant party if not the commit bit owner?
>>>>>> +1 for commit bit owners receiving automatic notification to review
>>>>>>
>>>>>> --
>>>>>> milan
>>>>>>
>>>>>> On Tue, Aug 14, 2018 at 12:56 AM, Daniel Alley 
>>>>>> wrote:
>>>>>>
>>>>>>> +1. My understanding is that this will not directly limit who can
>>>>>>> review or merge code, but should streamline the review process by 
>>>>>>> notifying
>>>>>>> relevant parties?
>>>>>>>
>>>>>>> On Mon, Aug 13, 2018 at 5:29 PM, David Davis 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We have come up with initial proposal of how to use code owners
>>>>>>>> feature in Pulp. Feedback on the initial proposal below is welcome. I 
>>>>>>>> will
>>>>>>>> try to collect the f

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 3:47 PM, David Davis  wrote:

> On Tue, Aug 14, 2018 at 9:35 AM Milan Kovacik  wrote:
>
>>
>>
>> On Tue, Aug 14, 2018 at 1:26 PM, David Davis 
>> wrote:
>>
>>> The relevant party could either be a subset of the commit bit owners
>>> (e.g. task group) or a set of people who don’t have the commit bit (e.g.
>>> QE). See the team examples from my original email.
>>>
>>
>>  So what you mean is actually a trusted subset of commit bit owners, like
>> the SMEs?
>>
>
> These teams aren’t necessarily a subset of commit bit owners but yes
> they’ll be subject matter experts (SMEs) for the code they own. Take QE for
> example. They might not have the commit bit to the pulp repo but they are
> still the SMEs for pulp_smash tests and thus they’ll probably be code
> owners for the smash tests in pulp and pulp_file.
>
>
>>  So we don't trust all commit bit owners equally when it comes to
>> particular git (sub)tree?
>>
>  And we trust (by blocking the merge) on e.g QE approving a PR more than
>> the commit bit owner that is outside of the subset?
>>  Or is it rather about decoupling the code review duty from the commit
>> bit ownership?
>>
>
> To answer these questions, I don’t think it’s about trust. It’s about (as
> you mention) decoupling merging code from code reviews. We want to make
> sure the appropriate people get notified and have a chance to review the
> PRs for which they are SMEs.
>

This has the same issue as the commit bit ownership, it's just more fine
grained and bound to particular subtrees: how does one become an SME?
What's the SME lifecycle?


>
>
>>  Why do we need commit bit owners then?
>>
>
> How else do we merge the code if no one has a commit bit?
>

  thru a bot for instance


>
>
>>
>> Cheers,
>> milan
>>
>>
>>>
>>> Daniel, you are correct. The only caveat is that PRs can’t be merged if
>>> they touch a file owned by a team and haven’t been approved by that team.
>>>
>>> David
>>>
>>>
>>> On Tue, Aug 14, 2018 at 6:35 AM Milan Kovacik 
>>> wrote:
>>>
>>>> +0 who's the relevant party if not the commit bit owner?
>>>> +1 for commit bit owners receiving automatic notification to review
>>>>
>>>> --
>>>> milan
>>>>
>>>> On Tue, Aug 14, 2018 at 12:56 AM, Daniel Alley 
>>>> wrote:
>>>>
>>>>> +1. My understanding is that this will not directly limit who can
>>>>> review or merge code, but should streamline the review process by 
>>>>> notifying
>>>>> relevant parties?
>>>>>
>>>>> On Mon, Aug 13, 2018 at 5:29 PM, David Davis 
>>>>> wrote:
>>>>>
>>>>>> We have come up with initial proposal of how to use code owners
>>>>>> feature in Pulp. Feedback on the initial proposal below is welcome. I 
>>>>>> will
>>>>>> try to collect the feedback and open a PUP by the end of the week. 
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> # Problem Statement
>>>>>>
>>>>>> For Pulp's review process, there are several areas we could improve:
>>>>>>
>>>>>> 1. It’s not always clear who should review files. Over time we have
>>>>>> developed subject matter experts for different areas of the codebase, but
>>>>>> these are not codified anywhere. It would be useful for us to define 
>>>>>> teams
>>>>>> need to review projects using code owners.
>>>>>>
>>>>>> 2. PRs go unnoticed. Github has a request-review feature, but only
>>>>>> members of the github organization can click 'request review' button. It
>>>>>> would be great if when a PR is opened people automatically received PR
>>>>>> review requests.
>>>>>>
>>>>>>
>>>>>> # Team Examples
>>>>>>
>>>>>> Functional Tests - The QE Team should be code owners of functional
>>>>>> tests that test core or core-maintained plugins
>>>>>> The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
>>>>>> this area
>>>>>>
>>>>>>
>>>>>> # Solution
>>>>>>
>>>>>> 1. Configure the code-owners feature of Github (
>

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

On Tue, Aug 14, 2018 at 1:26 PM, David Davis  wrote:

> The relevant party could either be a subset of the commit bit owners (e.g.
> task group) or a set of people who don’t have the commit bit (e.g. QE). See
> the team examples from my original email.
>

 So what you mean is actually a trusted subset of commit bit owners, like
the SMEs?
 So we don't trust all commit bit owners equally when it comes to
particular git (sub)tree?
 And we trust (by blocking the merge) on e.g QE approving a PR more than
the commit bit owner that is outside of the subset?
 Or is it rather about decoupling the code review duty from the commit bit
ownership?
 Why do we need commit bit owners then?

Cheers,
milan


>
> Daniel, you are correct. The only caveat is that PRs can’t be merged if
> they touch a file owned by a team and haven’t been approved by that team.
>
> David
>
>
> On Tue, Aug 14, 2018 at 6:35 AM Milan Kovacik  wrote:
>
>> +0 who's the relevant party if not the commit bit owner?
>> +1 for commit bit owners receiving automatic notification to review
>>
>> --
>> milan
>>
>> On Tue, Aug 14, 2018 at 12:56 AM, Daniel Alley  wrote:
>>
>>> +1. My understanding is that this will not directly limit who can review
>>> or merge code, but should streamline the review process by notifying
>>> relevant parties?
>>>
>>> On Mon, Aug 13, 2018 at 5:29 PM, David Davis 
>>> wrote:
>>>
>>>> We have come up with initial proposal of how to use code owners feature
>>>> in Pulp. Feedback on the initial proposal below is welcome. I will try to
>>>> collect the feedback and open a PUP by the end of the week. Thanks!
>>>>
>>>>
>>>> # Problem Statement
>>>>
>>>> For Pulp's review process, there are several areas we could improve:
>>>>
>>>> 1. It’s not always clear who should review files. Over time we have
>>>> developed subject matter experts for different areas of the codebase, but
>>>> these are not codified anywhere. It would be useful for us to define teams
>>>> need to review projects using code owners.
>>>>
>>>> 2. PRs go unnoticed. Github has a request-review feature, but only
>>>> members of the github organization can click 'request review' button. It
>>>> would be great if when a PR is opened people automatically received PR
>>>> review requests.
>>>>
>>>>
>>>> # Team Examples
>>>>
>>>> Functional Tests - The QE Team should be code owners of functional
>>>> tests that test core or core-maintained plugins
>>>> The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
>>>> this area
>>>>
>>>>
>>>> # Solution
>>>>
>>>> 1. Configure the code-owners feature of Github (
>>>> https://blog.github.com/2017-07-06-introducing-code-owners/). It will
>>>> allow a team of 2 or more people to be notified and asked for review when a
>>>> PR modifies a file within a certain area of the code.
>>>>
>>>> 2. Require code-owner review to merge. This is described in this
>>>> section: https://blog.github.com/2017-07-06-introducing-code-owners/
>>>> #an-extra-layer-of-code-security
>>>>
>>>>
>>>> # Process
>>>>
>>>> The code owner role is not related to the commit bit. It's designed to
>>>> get better reviews. Well reviewed work can be confidently merged by anyone
>>>> with the commit bit.
>>>>
>>>> To make a change to code owners, open a PR with the changes, and call
>>>> for a lazy consensus vote by mailing list. Similar to the PUP decision
>>>> making process, voting must be open for 10 days, and the committers of the
>>>> respective repository are voting.
>>>>
>>>> The code owners file itself should be owned by the core committers of
>>>> the repository.
>>>>
>>>>
>>>> ___
>>>> Pulp-dev mailing list
>>>> Pulp-dev@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>
>>>>
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp Code Owners

[Milan Kovacik]

+0 who's the relevant party if not the commit bit owner?
+1 for commit bit owners receiving automatic notification to review

--
milan

On Tue, Aug 14, 2018 at 12:56 AM, Daniel Alley  wrote:

> +1. My understanding is that this will not directly limit who can review
> or merge code, but should streamline the review process by notifying
> relevant parties?
>
> On Mon, Aug 13, 2018 at 5:29 PM, David Davis 
> wrote:
>
>> We have come up with initial proposal of how to use code owners feature
>> in Pulp. Feedback on the initial proposal below is welcome. I will try to
>> collect the feedback and open a PUP by the end of the week. Thanks!
>>
>>
>> # Problem Statement
>>
>> For Pulp's review process, there are several areas we could improve:
>>
>> 1. It’s not always clear who should review files. Over time we have
>> developed subject matter experts for different areas of the codebase, but
>> these are not codified anywhere. It would be useful for us to define teams
>> need to review projects using code owners.
>>
>> 2. PRs go unnoticed. Github has a request-review feature, but only
>> members of the github organization can click 'request review' button. It
>> would be great if when a PR is opened people automatically received PR
>> review requests.
>>
>>
>> # Team Examples
>>
>> Functional Tests - The QE Team should be code owners of functional tests
>> that test core or core-maintained plugins
>> The Tasking System  - bmbouter, daviddavis, and dalley are the SME in
>> this area
>>
>>
>> # Solution
>>
>> 1. Configure the code-owners feature of Github (
>> https://blog.github.com/2017-07-06-introducing-code-owners/). It will
>> allow a team of 2 or more people to be notified and asked for review when a
>> PR modifies a file within a certain area of the code.
>>
>> 2. Require code-owner review to merge. This is described in this section:
>> https://blog.github.com/2017-07-06-introducing-code-owners/#
>> an-extra-layer-of-code-security
>>
>>
>> # Process
>>
>> The code owner role is not related to the commit bit. It's designed to
>> get better reviews. Well reviewed work can be confidently merged by anyone
>> with the commit bit.
>>
>> To make a change to code owners, open a PR with the changes, and call for
>> a lazy consensus vote by mailing list. Similar to the PUP decision making
>> process, voting must be open for 10 days, and the committers of the
>> respective repository are voting.
>>
>> The code owners file itself should be owned by the core committers of the
>> repository.
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Revisit: sync modes

[Milan Kovacik]

On Thu, Aug 9, 2018 at 8:29 PM, Daniel Alley  wrote:

> It's possible we could want additional sync_modes in the future. To me,
>> sync mode deals with the contents of the repo during the sync. There are
>> other ways you would want to have a sync associate content with a
>> repository. Consider a retention behavior that retains 5 versions of each
>> unit, e.g. rpms, ansible modules, etc; that behavior is somewhere in
>> between mirror and additive. If we make mirror a boolean then to introduce
>> this retention feature we would have to add it as an additional option.
>> This creates the downside I hope to avoid which is that interaction between
>> options becomes complicated.
>>
>
>> For example, a call with both (mirror=False, retention=True) now becomes
>> more complicated to think about. Is it mirroring or using the retention
>> policy? How do these interact? At that point, it seems more complicated
>> than what we have now. The way to avoid this is by keeping them together as
>> one option, but that can only be done if it stays as a string.
>>
>
> These are all good points but I think "retention" would likely need to be
> a configurable parameter, probably one that you would have to pass in.  The
> default value could mean "unlimited retention", i.e.  "additive".
>
> So what you could do is:
>
> (mirror=False)   # this is normal additive mode,
>> retain everything.  let's say that default retention=0, which is
>> nonsensical and would map to this behavior instead
>>
> (mirror=False, retention=5) # retain at most 5 versions of any given
>> unit
>>
> (mirror=False, retention=1) # this is *almost* like mirror mode,
>> except that you would still keep one historical copy of units that are no
>> longer present in the upstream repository
>>
>
> Maybe it even makes sense to have retention be able to modify "mirror"
> mode, although this would make the concept of "mirror" more difficult to
> understand as you point out.  Maybe we could find a name that would be less
> misleading.
>
> (mirror=True, retention=5)   # retain at most 5 versions of any given
>> unit, *but purge units that that are no longer present in the upstream
>> repo entirely*
>>
>
> I don't have a specific use case in mind for that one, but maybe someone
> can think of one?
>

Anyone has a suggestion beyond 'mirror' and 'retention' vs. 'additive'
modes?
If we anticipate bigger sync mode variability  we should probably refactor
the DeclarativeVersion into a proper pipeline factory.
Btw I don't think it make sense to change the sync behaviour of a repo with
every sync call; I guess we'd better make the sync pipeline construction a
remote create time option.


>
> On Thu, Aug 9, 2018 at 12:53 PM, Brian Bouterse 
> wrote:
>
>> It's possible we could want additional sync_modes in the future. To me,
>> sync mode deals with the contents of the repo during the sync. There are
>> other ways you would want to have a sync associate content with a
>> repository. Consider a retention behavior that retains 5 versions of each
>> unit, e.g. rpms, ansible modules, etc; that behavior is somewhere in
>> between mirror and additive. If we make mirror a boolean then to introduce
>> this retention feature we would have to add it as an additional option.
>> This creates the downside I hope to avoid which is that interaction between
>> options becomes complicated.
>>
>> For example, a call with both (mirror=False, retention=True) now becomes
>> more complicated to think about. Is it mirroring or using the retention
>> policy? How do these interact? At that point, it seems more complicated
>> than what we have now. The way to avoid this is by keeping them together as
>> one option, but that can only be done if it stays as a string.
>>
>> On Thu, Aug 9, 2018 at 9:04 AM, Milan Kovacik 
>> wrote:
>>
>>>
>>>
>>> On Wed, Aug 8, 2018 at 7:54 PM, Jeff Ortel  wrote:
>>>
>>>> I'm not convinced that *named* sync mode is a good approach.  I doubt
>>>> it will ever be anything besides (additive|mirror) which really boils down
>>>> to mirror (or not).  Perhaps the reasoning behind a *named* mode is
>>>> that it is potentially more extensible in that the API won't be impacted
>>>> when a new mode is needed.  The main problem with this approach is that the
>>>> mode names are validated and interpreted in multiple places. Adding another
>>>> mode will require coordinated changes in both the core and most plugins.
>&g

Re: [Pulp-dev] Revisit: sync modes

[Milan Kovacik]

On Wed, Aug 8, 2018 at 7:54 PM, Jeff Ortel  wrote:

> I'm not convinced that *named* sync mode is a good approach.  I doubt it
> will ever be anything besides (additive|mirror) which really boils down to
> mirror (or not).  Perhaps the reasoning behind a *named* mode is that it
> is potentially more extensible in that the API won't be impacted when a new
> mode is needed.  The main problem with this approach is that the mode names
> are validated and interpreted in multiple places. Adding another mode will
> require coordinated changes in both the core and most plugins.  Generally,
> I'm an advocate of named things like *modes* and *policies* but given the
> orthogonal nature of the two modes we currently support *and* that no
> *real* or anticipated use cases for additional modes are known, I'm not
> convinced it's a good fit.  Are there any *real* or anticipated use cases
> I'm missing?
>

Looking at the code[1] we're actually talking about almost a (pipeline)
factory that has exactly 2 modes of operation with a limited possibilities
of extending, unsure that the possibility to extend was a goal though.
Moreover it turns out current implementation prevents using (class-level)
constants instead of custom strings due to plugin--platform import issues:
core serializer can't refer to DeclarativeVersion.defaul_sync_mode --- at
least I wasn't able to make this work as part of the sync_mode docstring
PR[2] review suggestion.


>
> I propose we replace the (str)sync_mode="" with (bool)mirror=False
> anywhere stored or passed.
>
> Are there any *real* or anticipated use cases I'm missing?
>
> Thoughts?
>

I'm afraid replacing custom strings with True/False won't make the
situation much better.
I'd vote for some refactor besides other things, it might better be part of
remote (or repository) creation endpoint.

Cheers,
milan

[1]
https://github.com/pulp/pulp/blob/master/plugin/pulpcore/plugin/stages/declarative_version.py#L100#L114
[2] https://github.com/pulp/pulp/pull/3583#discussion_r208869824



>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 'id' versus 'pulp_id' on Content

[Milan Kovacik]

On Tue, Aug 7, 2018 at 6:47 PM, Jeff Ortel  wrote:

> After long consideration, I have had a change of heart about this.  I
> think.  In short, Pulp's data model has unique requirements that make it
> acceptable to deviate from common convention regarding ID as the PK.
> Mainly that the schema is extensible by plugin writers.  Given the plugin
> architecture, I think it's reasonable to think of "core" fields like: ID,
> CREATED and LAST_MODIFIED as metadata.  Although, the ID is harder to fit
> in this semantic, I think it's reasonable to do for consistency and to
> support the user query use-case re: content having an natural ID
> attribute.  Taking this further, the *href* attributes *could* be though
> of in the same category.
>
> With this in mind, I'm thinking that the leading underscore (_) could be
> used broadly to denote *generated* *or metadata* fields and the following
> would be reasonable:
>
> _id
> _created
> _last_updated
>
> _href
> _added_href
> _removed_href
> _content_href
>
> I highly value consistency so this applies to the entire schema.
>

I'd even go as far as making a distinct core attribute of a plug-in
provided model: rpm.core.id, rpm.core.href, namespacing the model and
promoting composition.
This would have the benefit of being able to easily distinguish between
platform and plug-in specific data especially if plug-in specific data is
expected to be the main user interaction site i.e user_importance(rpm.id) >
user_importance(rpm.core.id).

 --
milan


> This will introduce a few fairly odd things into the schema that I
> *suppose* we can live with.
>
> - Two fields on *some* tables named (ID ,  _ID).  To mitigate confusion,
> we should serialize the *pk* and not* _id*.  This will also be consistent
> with *pk* parameters passed in.
> - I expect django will generate foreign key fields with double
> understores.  Eg: content__id
>

> I'm still -1 for using a *pulp_* prefix.
>
> Thoughts?
>
>
> On 06/18/2018 01:15 PM, Daniel Alley wrote:
>
> I'm -1 on going the underscore idea, partly because of the aforementioned
> confusion issue, but also partly because but I've noticed that in our API,
> the "underscore" basically has a semantic meeting of "href, [which is]
> generated on the fly, not stored in the db".
>
> Specifically:
>
>- '_href'
>- '_added_href'
>- '_removed_href'
>- '_content_href'
>
> So I think if we use a prefix, we should avoid using one that already has
> a semantic meaning (I don't know whether we actually planned for that to be
> the case, but I think it's a useful pattern / distinction and I don't think
> we should mess with it).
>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] CI not working on RHEL7 (for pulp_rpm)

[Milan Kovacik]

Folks,

It seems to me the F26 CI is still broken:
https://paste.fedoraproject.org/paste/RVrMq9GoBbiFfbqHVm03mA
https://pulpadmin.fedorapeople.org/jenkins/jobs/unittest-pulp_rpm-pr/builds/1155/node-type=f26-np.txt
But do we need F26? It's been unmaintained IIRC for about a month already.

--
milan

On Wed, Jul 11, 2018 at 9:53 PM, Milan Kovacik  wrote:

> David, Patrick
>
> thanks for handling this!
>
> --
> milan
>
> On Wed, Jul 11, 2018 at 9:10 PM, David Davis 
> wrote:
>
>> @milan, I thought you were gone for the day so I went ahead and opened a
>> PR to package libsolv:
>>
>> https://github.com/pulp/pulp-packaging/pull/47
>> <https://github.com/pulp/pulp-packaging/pull/47/commits>
>>
>> After it’s merged, I think RHEL should be good and hopefully your PR will
>> pass as well.
>>
>> David
>>
>>
>> On Wed, Jul 11, 2018 at 2:35 PM Milan Kovacik 
>> wrote:
>>
>>> Thanks for the feedback guys!
>>>
>>> @David, what in particular needs doing to carry the libsolv in Pulp
>>> repos?
>>>
>>> Cheers,
>>> milan
>>>
>>> On Wed, Jul 11, 2018 at 7:52 PM, David Davis 
>>> wrote:
>>>
>>>> I agree we should carry the dependency until it’s released. The RHEL7
>>>> breakage is blocking QE currently so let’s revert the change[0] while
>>>> @milan and I can work on getting a PR open to pulp-packaging to get the
>>>> dependency into our repos.
>>>>
>>>> [0] https://github.com/pulp/pulp-packaging/pull/46
>>>>
>>>> David
>>>>
>>>>
>>>> On Wed, Jul 11, 2018 at 1:14 PM Patrick Creech 
>>>> wrote:
>>>>
>>>>> On Wed, 2018-07-11 at 18:33 +0200, Milan Kovacik wrote:
>>>>> > Folks,
>>>>> >
>>>>> > We've merged a package dependency requirement update[1] to make it
>>>>> possible for the rich-dependencies work to be mergeable[2].
>>>>> >
>>>>> > This has broken the EL7 builds
>>>>>
>>>>> So, often times we have needs to carry new/updated dependencies that
>>>>> aren't in an EL7 repo proper, and also not in the EPEL repos for
>>>>> maintaining our EL7 support.
>>>>>
>>>>> I do not think it'll be tenable to have EL7 support broken for any
>>>>> significant period of time.  Perhaps some discussions need to happen to
>>>>> have an updated libsolv carried in pulp's repos for some time
>>>>> untill a suitable version lands in a dependent repo.
>>>>>
>>>>> > Fedora 27&28 (stable) isn't affected.
>>>>> >
>>>>> > Cheers,
>>>>> > milan
>>>>> >
>>>>> > [1] https://github.com/pulp/pulp-packaging/pull/45
>>>>> > [2] https://github.com/pulp/pulp_rpm/pull/1122
>>>>> > ___
>>>>> > Pulp-dev mailing list
>>>>> > Pulp-dev@redhat.com
>>>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>> ___
>>>>> Pulp-dev mailing list
>>>>> Pulp-dev@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>>>
>>>>
>>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] CI not working on RHEL7 (for pulp_rpm)

[Milan Kovacik]

Folks,

We've merged a package dependency requirement update[1] to make it possible
for the rich-dependencies work to be mergeable[2].

This has broken the RHEL7 builds, which was expected and there's a bug to
track the update of RHEL7 with that regard[3], scheduled for RHEL7.7

Fedora 27&28 (stable) isn't affected.

Cheers,
milan

[1] https://github.com/pulp/pulp-packaging/pull/45
[2] https://github.com/pulp/pulp_rpm/pull/1122
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1596238
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Revising PUPs

[Milan Kovacik]

Hey David,

thanks, +1

--
milan

On Mon, Jul 9, 2018 at 1:49 PM, David Davis  wrote:

> I’ve opened a PR with the process on how to revise a PUP. Reviews/feedback
> are welcome:
>
> https://github.com/pulp/pups/pull/11
>
> I’d also like to call a vote on this proposed change. Here’s the voting
> model from PUP-1:
>
> +1: "Will benefit the project and should definitely be adopted."
> +0: "Might benefit the project and is acceptable."
> -0: "Might not be the right choice but is acceptable."
> -1: "I have serious reservations that need to be thought through and
> addressed."
>
> Deadline will be July 22, 2018.
>
> David
>
>
> On Tue, Jul 3, 2018 at 11:14 AM David Davis  wrote:
>
>> While there is a process for revising PUPs before they are accepted[0],
>> we don’t have any process for revising PUPs after they are accepted. I’d
>> like to upate PUP-1[1] to create a simple but formal process for revising
>> accepted PUPs.
>>
>> I was thinking we should add a section (“Revising an Accepted PUP”) that
>> says say revising a PUP follows the same process as creating a new PUP.
>> This includes an initial discussion period followed by a PR against the PUP
>> with the proposed change. After that, there should be a vote decided by our
>> existing lazy consensus model.
>>
>> Thoughts?
>>
>> [0] https://github.com/pulp/pups/blob/master/pup-0001.md#revision
>> [1] https://github.com/pulp/pups/blob/master/pup-0001.md
>>
>> David
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp RPM dependency solver refactoring dilemma

[Milan Kovacik]

On Tue, Jun 26, 2018 at 8:04 PM, Justin Sherrill 
wrote:

>
>
> On 06/26/2018 11:30 AM, Milan Kovacik wrote:
>
>> Folks,
>>
>> TL;DR should we support alternative solvers (configuration) during
>> recursive unit association?
>>
>> I've been refactoring the current approach to RPM dependency solving
>> for e.g the recursive copy to be able to handle rich dependencies[1].
>>
>> While testing, I ran into an dependency issue that is caused by me not
>> processing file-provides records correctly[2].
>>
>> No matter the current insufficiency in my coding, a user trying to
>> copy stuff from a repo with libsolv-unresolvable dependencies might
>> hit similar issues and consider them regressions from previous
>> behavior, hence the question:
>>
>> Should the user be able to select a solver (configuration) for
>> particular associate call thru the REST API?
>>
> I commented on the PR, but i think the behavior we're seeing is okay and
> can be ignored (assuming we can still pull in the deps that are
> available).  Assuming we can, do we still need it to be configurable?
>

Turns out we can by simply ignoring the solver problems; might have as well
investigated it yesterday ;)


> I would also like to point out this issue to keep in mind:
> https://pulp.plan.io/issues/2478


That behaviour is already addressed in the PR by the solver considering the
target repository content
<https://github.com/pulp/pulp_rpm/pull/1122/files#diff-9c78b41ed1e2ec586589ed71d730b93eR56>
.
I wonder whether I should set the issue state accordingly...


>
>
> Justin
>
>
>
>> Cheers,
>> milan
>>
>>
>> [1] https://github.com/pulp/pulp_rpm/pull/1122
>> [2] https://github.com/pulp/pulp_rpm/pull/1122#issuecomment-400061802
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Pulp RPM dependency solver refactoring dilemma

[Milan Kovacik]

Folks,

TL;DR should we support alternative solvers (configuration) during
recursive unit association?

I've been refactoring the current approach to RPM dependency solving
for e.g the recursive copy to be able to handle rich dependencies[1].

While testing, I ran into an dependency issue that is caused by me not
processing file-provides records correctly[2].

No matter the current insufficiency in my coding, a user trying to
copy stuff from a repo with libsolv-unresolvable dependencies might
hit similar issues and consider them regressions from previous
behavior, hence the question:

Should the user be able to select a solver (configuration) for
particular associate call thru the REST API?

Cheers,
milan


[1] https://github.com/pulp/pulp_rpm/pull/1122
[2] https://github.com/pulp/pulp_rpm/pull/1122#issuecomment-400061802

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] 'id' versus 'pulp_id' on Content

[Milan Kovacik]

uuid will most likely hit the same issue one day.

I wonder whether we're not forcing (model) inheritance where
composition might have been a better fit; how about pulp core
providing some sort of a (meta) container object that:
* holds the real plugin content unit
* is used to provide the core functionality (versioned repos etc)...
* queries to the inside of the container will be namespaced; e.g
GET@.../errata/content/pulp_meta_last_updated__gt=yesterday__errata_id=foo

--
milan

On Thu, Jun 14, 2018 at 9:38 PM, Jeff Ortel  wrote:
>
>
> On 06/14/2018 12:19 PM, Jeff Ortel wrote:
>>
>>
>>
>> On 06/14/2018 10:37 AM, Daniel Alley wrote:
>>>
>>> I will make one more suggestion.  What about naming "id" -> "uuid"?  This
>>> carries the clear connotation that it is a unique identifier so it is less
>>> likely to be confusing a la "id and _id", and is still less likely to have a
>>> namespace conflict.
>>
>>
>> Appreciate the suggestion but this would only be marginally less
>> confusing.
>
>
> Reconsidering this suggestion for the reasons you outlined.
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Lazy for Pulp3

[Milan Kovacik]

On Thu, May 31, 2018 at 11:39 PM, Brian Bouterse  wrote:
> I updated the epic (https://pulp.plan.io/issues/3693) to use this new
> language.
>
> policy=immediate  -> downloads now while the task runs (no lazy). Also the
> default if unspecified.
> policy=cache-and-save   -> All the steps in the diagram. Content that is
> downloaded is saved so that it's only ever downloaded once.
> policy=cache -> All the steps in the diagram except step 14. If squid
> pushes the bits out of the cache, it will be re-downloaded again to serve to
> other clients requesting the same bits.
>
> Also @milan, see inline for answers to your question.
>
> On Wed, May 30, 2018 at 3:48 PM, Milan Kovacik  wrote:
>>
>> On Wed, May 30, 2018 at 4:50 PM, Brian Bouterse 
>> wrote:
>> >
>> >
>> > On Wed, May 30, 2018 at 8:57 AM, Tom McKay 
>> > wrote:
>> >>
>> >> I think there is a usecase for "proxy only" like is being described
>> >> here.
>> >> Several years ago there was a project called thumbslug[1] that was used
>> >> in a
>> >> version of katello instead of pulp. It's job was to check entitlements
>> >> and
>> >> then proxy content from a cdn. The same functionality could be
>> >> implemented
>> >> in pulp. (Perhaps it's even as simple as telling squid not to cache
>> >> anything
>> >> so the content would never make it from cache to pulp in current
>> >> pulp-2.)
>> >
>> >
>> > What would you call this policy?
>> > policy=proxy?
>> > policy=stream-dont-save?
>> > policy=stream-no-save?
>> >
>> > Are the names 'on-demand' and 'immediate' clear enough? Are there better
>> > names?
>> >>
>> >>
>> >> Overall I'm +1 to the idea of an only-squid version, if others think it
>> >> would be useful.
>> >
>> >
>> > I understand describing this as a "only-squid" version, but for clarity,
>> > the
>> > streamer would still be required because it is what requests the bits
>> > with
>> > the correctly configured downloader (certs, proxy, etc). The streamer
>> > streams the bits into squid which provides caching and client
>> > multiplexing.
>>
>> I have to admit it's just now I'm reading
>>
>> https://docs.pulpproject.org/dev-guide/design/deferred-download.html#apache-reverse-proxy
>> again because of the SSL termination. So the new plan is to use the
>> streamer to terminate the SSL instead of the Apache reverse proxy?
>
>
> The plan for right now is to not use a reverse proxy and have the client's
> connection terminate at squid directly either via http or https depending on
> how squid is configured. The Reverse proxy in pulp2's design served to
> validate the signed urls and rewrite them for squid. This first
> implementation won't use signed urls. I believe that means we don't need a
> reverse proxy here yet.

I don't think I understand; so Squid will be used to terminate TLS but
it won't be used as a reverse proxy?




>
>>
>> W/r the construction of the URL of an artifact, I thought it would be
>> stored in the DB, so the Remote would create it during the sync.
>
>
> This is correct. The inbound URL from the client after the redirect will
> still be a reference that the "Pulp content app" will resolve to a
> RemoteArtifact. Then the streamer will use that RemoteArtifact data to
> correctly build the downloader. That's the gist of it at least.



>
>>
>> >
>> > To confirm my understanding this "squid-only" policy would be the same
>> > as
>> > on-demand except that it would *not* perform step 14 from the diagram
>> > here
>> > (https://pulp.plan.io/issues/3693). Is that right?
>> yup
>> >
>> >>
>> >>
>> >> [1] https://github.com/candlepin/thumbslug
>> >>
>> >> On Wed, May 30, 2018 at 8:34 AM, Milan Kovacik 
>> >> wrote:
>> >>>
>> >>> On Tue, May 29, 2018 at 9:31 PM, Dennis Kliban 
>> >>> wrote:
>> >>> > On Tue, May 29, 2018 at 11:42 AM, Milan Kovacik
>> >>> > 
>> >>> > wrote:
>> >>> >>
>> >>> >> On Tue, May 29, 2018 at 5:13 PM, Dennis Kliban 
>> >>> >> wrote:
>> >>> >> > On Tue, May 29, 2018 at 10:41 AM, Milan Kovacik
>> >>> >> > 
>> >>> >> > wrote:
>>

Re: [Pulp-dev] Lazy for Pulp3

[Milan Kovacik]

On Wed, May 30, 2018 at 4:50 PM, Brian Bouterse  wrote:
>
>
> On Wed, May 30, 2018 at 8:57 AM, Tom McKay  wrote:
>>
>> I think there is a usecase for "proxy only" like is being described here.
>> Several years ago there was a project called thumbslug[1] that was used in a
>> version of katello instead of pulp. It's job was to check entitlements and
>> then proxy content from a cdn. The same functionality could be implemented
>> in pulp. (Perhaps it's even as simple as telling squid not to cache anything
>> so the content would never make it from cache to pulp in current pulp-2.)
>
>
> What would you call this policy?
> policy=proxy?
> policy=stream-dont-save?
> policy=stream-no-save?
>
> Are the names 'on-demand' and 'immediate' clear enough? Are there better
> names?
>>
>>
>> Overall I'm +1 to the idea of an only-squid version, if others think it
>> would be useful.
>
>
> I understand describing this as a "only-squid" version, but for clarity, the
> streamer would still be required because it is what requests the bits with
> the correctly configured downloader (certs, proxy, etc). The streamer
> streams the bits into squid which provides caching and client multiplexing.

I have to admit it's just now I'm reading
https://docs.pulpproject.org/dev-guide/design/deferred-download.html#apache-reverse-proxy
again because of the SSL termination. So the new plan is to use the
streamer to terminate the SSL instead of the Apache reverse proxy?

W/r the construction of the URL of an artifact, I thought it would be
stored in the DB, so the Remote would create it during the sync.

>
> To confirm my understanding this "squid-only" policy would be the same as
> on-demand except that it would *not* perform step 14 from the diagram here
> (https://pulp.plan.io/issues/3693). Is that right?
yup
>
>>
>>
>> [1] https://github.com/candlepin/thumbslug
>>
>> On Wed, May 30, 2018 at 8:34 AM, Milan Kovacik 
>> wrote:
>>>
>>> On Tue, May 29, 2018 at 9:31 PM, Dennis Kliban 
>>> wrote:
>>> > On Tue, May 29, 2018 at 11:42 AM, Milan Kovacik 
>>> > wrote:
>>> >>
>>> >> On Tue, May 29, 2018 at 5:13 PM, Dennis Kliban 
>>> >> wrote:
>>> >> > On Tue, May 29, 2018 at 10:41 AM, Milan Kovacik
>>> >> > 
>>> >> > wrote:
>>> >> >>
>>> >> >> Good point!
>>> >> >> More the second; it might be a bit crazy to utilize Squid for that
>>> >> >> but
>>> >> >> first, let's answer the why ;)
>>> >> >> So why does Pulp need to store the content here?
>>> >> >> Why don't we point the users to the Squid all the time (for the
>>> >> >> lazy
>>> >> >> repos)?
>>> >> >
>>> >> >
>>> >> > Pulp's Streamer needs to fetch and store the content because that's
>>> >> > Pulp's
>>> >> > primary responsibility.
>>> >>
>>> >> Maybe not that much the storing but rather the content views
>>> >> management?
>>> >> I mean the partitioning into repositories, promoting.
>>> >>
>>> >
>>> > Exactly this. We want Pulp users to be able to reuse content that was
>>> > brought in using the 'on_demand' download policy in other repositories.
>>> I see.
>>>
>>> >
>>> >>
>>> >> If some of the content lived in Squid and some lived
>>> >> > in Pulp, it would be difficult for the user to know what content is
>>> >> > actually
>>> >> > available in Pulp and what content needs to be fetched from a remote
>>> >> > repository.
>>> >>
>>> >> I'd say the rule of the thumb would be: lazy -> squid, regular -> pulp
>>> >> so not that difficult.
>>> >> Maybe Pulp could have a concept of Origin, where folks upload stuff to
>>> >> a Pulp repo, vs. Proxy for it's repo storage policy?
>>> >>
>>> >
>>> > Squid removes things from the cache at some point. You can probably
>>> > configure it to never remove anything from the cache, but then we would
>>> > need
>>> > to implement orphan cleanup that would work across two systems: pulp
>>> > and
>>> > squid.
>>>
>>> Actually "remote" units wouldn't need orphan cleaning from the disk,
>>

Re: [Pulp-dev] Lazy for Pulp3

[Milan Kovacik]

On Tue, May 29, 2018 at 9:31 PM, Dennis Kliban  wrote:
> On Tue, May 29, 2018 at 11:42 AM, Milan Kovacik  wrote:
>>
>> On Tue, May 29, 2018 at 5:13 PM, Dennis Kliban  wrote:
>> > On Tue, May 29, 2018 at 10:41 AM, Milan Kovacik 
>> > wrote:
>> >>
>> >> Good point!
>> >> More the second; it might be a bit crazy to utilize Squid for that but
>> >> first, let's answer the why ;)
>> >> So why does Pulp need to store the content here?
>> >> Why don't we point the users to the Squid all the time (for the lazy
>> >> repos)?
>> >
>> >
>> > Pulp's Streamer needs to fetch and store the content because that's
>> > Pulp's
>> > primary responsibility.
>>
>> Maybe not that much the storing but rather the content views management?
>> I mean the partitioning into repositories, promoting.
>>
>
> Exactly this. We want Pulp users to be able to reuse content that was
> brought in using the 'on_demand' download policy in other repositories.
I see.

>
>>
>> If some of the content lived in Squid and some lived
>> > in Pulp, it would be difficult for the user to know what content is
>> > actually
>> > available in Pulp and what content needs to be fetched from a remote
>> > repository.
>>
>> I'd say the rule of the thumb would be: lazy -> squid, regular -> pulp
>> so not that difficult.
>> Maybe Pulp could have a concept of Origin, where folks upload stuff to
>> a Pulp repo, vs. Proxy for it's repo storage policy?
>>
>
> Squid removes things from the cache at some point. You can probably
> configure it to never remove anything from the cache, but then we would need
> to implement orphan cleanup that would work across two systems: pulp and
> squid.

Actually "remote" units wouldn't need orphan cleaning from the disk,
just dropping them from the DB would suffice.

>
> Answering that question would still be difficult. Not all content that is in
> the repository that was synced using on_demand download policy will be in
> Squid - only the content that has been requested by clients. So it's still
> hard to know which of the content units have been downloaded and which have
> not been.

But the beauty is exactly in that: we don't have to track whether the
content is downloaded if it is reverse-proxied[1][2].
Moreover, this would work both with and without a proxy between Pulp
and the Origin of the remote unit.
A "remote" content artifact might just need to carry it's URL in a DB
column for this to work; so the async artifact model, instead of the
"policy=on-demand"  would have a mandatory remote "URL" attribute; I
wouldn't say it's more complex than tracking the "policy" attribute.

>
>
>>
>> >
>> > As Pulp downloads an Artifact, it calculates all the checksums and it's
>> > size. It then performs validation based on information that was provided
>> > from the RemoteArtifact. After validation is performed, the Artifact, is
>> > saved to the database and it's final place in
>> > /var/lib/content/artifacts/.
>>
>> This could be still achieved by storing the content just temporarily
>> in the Squid proxy i.e use Squid as the content source, not the disk.
>>
>> > Once this information is in the database, Pulp's web server can serve
>> > the
>> > content without having to involve the Streamer or Squid.
>>
>> Pulp might serve just the API and the metadata, the content might be
>> redirected to the Proxy all the time, correct?
>> Doesn't Crane do that btw?
>
>
> Theoretically we could do this, but in practice we would run into problems
> when we needed to scale out the Content app. Right now when the Content app
> needs to be scaled, a user can launch another machine that will run the
> Content app. Squid does not support that kind of scaling. Squid can only
> take advantage of additional cores in a single machine

I don't think I understand; proxies are actually designed to scale[1]
and are used as tools to scale the web too.

This is all about the How question but when it comes to my original
Why, please correct me if I'm being wrong, the answer so far has been:
 Pulp always downloads the content because that's what it is supposed to do.

Cheers,
milan

[1] https://en.wikipedia.org/wiki/Reverse_proxy
[2] https://paste.fedoraproject.org/paste/zkBTyxZjm330FsqvPP0lIA
[3] 
https://wiki.squid-cache.org/Features/CacheHierarchy?highlight=%28faqlisted.yes%29

>
>>
>>
>> Cheers,
>> milan
>>
>> >
>> > -dennis
>> >
>> >
>> >
>> >
>> &

Re: [Pulp-dev] Lazy for Pulp3

[Milan Kovacik]

Good point!
More the second; it might be a bit crazy to utilize Squid for that but
first, let's answer the why ;)
So why does Pulp need to store the content here?
Why don't we point the users to the Squid all the time (for the lazy repos)?

--
cheers
milan

On Tue, May 29, 2018 at 4:25 PM, Brian Bouterse  wrote:
>
> On Mon, May 28, 2018 at 9:57 AM, Milan Kovacik  wrote:
>>
>> Hi,
>>
>> Looking at the diagram[1] I'm wondering what's the reasoning behind
>> Pulp having to actually fetch the content locally?
>
>
> Is the question "why is Pulp doing the fetching and not Squid?" or "why is
> Pulp storing the content after fetching it?" or both?
>
>> Couldn't Pulp just rely on the proxy with regards to the content
>> streaming?
>>
>> Thanks,
>> milan
>>
>>
>> [1] https://pulp.plan.io/attachments/130957
>>
>> On Fri, May 25, 2018 at 9:11 PM, Brian Bouterse 
>> wrote:
>> > A mini-team of core devs** met to talk through lazy use cases for Pulp3.
>> > It's effectively the same lazy from Pulp2 except:
>> >
>> > * it's now built into core (not just RPM)
>> > * It disincludes repo protection use cases because we haven't added repo
>> > protection to Pulp3 yet
>> > * It disincludes the "background" policy which based on feedback from
>> > stakeholders provided very little value
>> > * it will no longer will depend on Twisted as a dependency. It will use
>> > asyncio instead.
>> >
>> > While it is being built into core, it will require minimal support by a
>> > plugin writer to add support for it. Details in the epic below.
>> >
>> > The current use cases along with a technical plan are written on this
>> > epic:
>> > https://pulp.plan.io/issues/3693
>> >
>> > We're putting it out for comment, questions, and feedback before we
>> > start
>> > into the code. I hope we are able to add this into our next sprint.
>> >
>> > ** ipanova, jortel, ttereshc, dkliban, bmbouter
>> >
>> > Thanks!
>> > Brian
>> >
>> >
>> > ___
>> > Pulp-dev mailing list
>> > Pulp-dev@redhat.com
>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>> >
>
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Content types which are not compatible with the normal pulp workflow

[Milan Kovacik]

Thanks for the explanation!

On Mon, May 28, 2018 at 4:17 PM, Daniel Alley <dal...@redhat.com> wrote:
>> Is that because of the rollback actually creates version #3 that's
>> "newer" but lacks the rolled-back commits?
>> So there are some "merge" conflict if folks, that cloned #2, want to
>> pull from version #3 but their branch contains a commit the origin
>> lacks now?
>> Or rather that the published bits of the version #2 doesn't exist
>> anymore at all?
>
>
> The first one.  It would be like if someone force-pushed to the git
> repository, removing the last couple of commits of history.  It's basically
> the same problem.
>
>> Does it mean a publication directory git tree is built anew every time
>> a rollback happens?
>
>
> What it would have to do is take the existing git tree and apply new commits
> on top to return the contents of the repository to the state you want to
> roll it back to.
>
>> So Pulp history and the original project history are meant to be
>> different?
>> Can there be ever conflicts?
>
>
> It's not that they're meant to be different, but I think it is an
> unavoidable problem if you want to do rollbacks in Pulp.
>
> The source git repository for the project, whether it's on github or the
> admin's machine, is separate from Pulp's copy. The second you add a commit
> to one and not the other (by doing rollback w/ linear git history from the
> client's perspective), the histories will diverge.  It's unavoidable, that's
> just how git works.  You can keep the content of the files in the repo
> identical but the history will never be equivalent again.

...impairing the usability of Pulp as the "master" repository

>
> Basically, it is mutually exclusive to have:
>
> * Pulp not be the "master" git repository e.g. the admin is syncing /
> uploading it from somewhere else
> * maintain linear git history
> * be able to do rollbacks in Pulp
> * keep identical git history between Pulp and the git repository being
> synced / uploaded into Pulp
>
> One of them has to give.

+1

I believe any content type/plug-in with its own idea of  content
versioning will have the same conflict.
Wrapping/translating from content-specific versioning scheme to Pulp
versioning scheme sounds like a headache even if Pulp supports a
non-linear history one day.

Let's forget about it and give the plug-in the ability to opt-out from
the core versioning scheme instead?

Cheers,
milan

>
>
> On Mon, May 28, 2018 at 8:01 AM, Milan Kovacik <mkova...@redhat.com> wrote:
>>
>> On Sat, May 26, 2018 at 2:23 AM, Daniel Alley <dal...@redhat.com> wrote:
>> > @Brian
>> >
>> > I agree with a lot of those points, but I would say that we're not just
>> > competing against hodgepodge collections of "scripts", but also against
>> > writing small microservice-y Flask apps that only implement the API for
>> > one
>> > content type.
>> >
>> > Also, rollback is not something Pulp would necessarily be able to offer
>> > with
>> > respect to history-sensitive content and metadata, like git
>> > repositories, or
>> > the Cargo example I provided.  It's still something the plugin writer
>> > would
>> > have to implement themselves in this case.
>> >
>> > @Jeff
>> >
>> >> perhaps a new component of a Publication like PublishedDirectory that
>> >> references an OSTree/Git repository created in /var/lib/pulp/published.
>> >
>> >
>> > I like the idea generally, but I don't think it would be able to be a
>> > component of a Publication.  I think it would need to be an alternative
>> > to a
>> > Publication which fulfills a similar function.
>> >
>> > The fundamental problem is this scenario:
>> >
>> > You upload a git repository with a git repository plugin
>> > You publish and distribute version 1 of the git repository
>> > You publish and distribute version 2 of the git repository
>> > A client downloads the git repository
>> > You notice a problem and decide to roll back to version 1.  A
>> > publication of
>> > version 1 already exists, which you distribute.
>> > Clients have a broken git history.  New clients can download the old
>> > version
>> > but anyone who has already downloaded version 2 will not be able to roll
>> > back to version 1 by pulling from Pulp
>>
>> Just trying to understand the situation:
>> Is that because of the rollback actually creates version #3 that's
>> "newer" 

Re: [Pulp-dev] Content types which are not compatible with the normal pulp workflow

[Milan Kovacik]

On Sat, May 26, 2018 at 2:23 AM, Daniel Alley  wrote:
> @Brian
>
> I agree with a lot of those points, but I would say that we're not just
> competing against hodgepodge collections of "scripts", but also against
> writing small microservice-y Flask apps that only implement the API for one
> content type.
>
> Also, rollback is not something Pulp would necessarily be able to offer with
> respect to history-sensitive content and metadata, like git repositories, or
> the Cargo example I provided.  It's still something the plugin writer would
> have to implement themselves in this case.
>
> @Jeff
>
>> perhaps a new component of a Publication like PublishedDirectory that
>> references an OSTree/Git repository created in /var/lib/pulp/published.
>
>
> I like the idea generally, but I don't think it would be able to be a
> component of a Publication.  I think it would need to be an alternative to a
> Publication which fulfills a similar function.
>
> The fundamental problem is this scenario:
>
> You upload a git repository with a git repository plugin
> You publish and distribute version 1 of the git repository
> You publish and distribute version 2 of the git repository
> A client downloads the git repository
> You notice a problem and decide to roll back to version 1.  A publication of
> version 1 already exists, which you distribute.
> Clients have a broken git history.  New clients can download the old version
> but anyone who has already downloaded version 2 will not be able to roll
> back to version 1 by pulling from Pulp

Just trying to understand the situation:
Is that because of the rollback actually creates version #3 that's
"newer" but lacks the rolled-back commits?
So there are some "merge" conflict if folks, that cloned #2, want to
pull from version #3 but their branch contains a commit the origin
lacks now?
Or rather that the published bits of the version #2 doesn't exist
anymore at all?

>
> We need to prevent step 5 from happening.
>
> There are a couple of possible solutions to this problem:
>
> As a Pulp admin, you ignore Pulp's rollback functionality.  Instead of using
> Pulp to roll back, you manually revert the commits using git, and upload a
> new version of the repository to Pulp as "version 3".  You then distribute
> version 3 instead of version 1.  You understand that if you were to publish
> and old version using Pulp, it would misbehave for clients that tried to
> pull / update instead of cloning.

In my opinion folks needing Pulp to track a git(-like) repo are
probably interested in more workflows than just the clone.

>
> As a Pulp admin / plugin writer / user, you know that the client for the
> content type will never try to pull or update, only clone.  Therefore it is
> not a problem for you and can be ignored.

The cloning might be equivalent of just snapshotting the tree at a
particular commit and just publishing a plain tar.gz w/o the git
structures.
Limiting but clean?

>
> As a Plugin writer, whenever you publish a new version of the git
> repository, you delete or invalidate every publication for previous versions
> for the distribution base path.  If a Pulp admin wants to roll back, they
> need to create a new Publication.  The Plugin knows to apply revert commits
> on top of the repository to keep history linear.
>
> But really we've just pushed the problem forwards.  What happens when you
> want to upload future versions?  Now history of the git repository in Pulp
> is different from the Pulp admin's git repo history
> This is only acceptable for content types where the history is immaterial to
> the content itself. Probably viable for Cargo, but probably not a Git
> content type.
>

Does it mean a publication directory git tree is built anew every time
a rollback happens?
So Pulp history and the original project history are meant to be different?
Can there be ever conflicts?


> As a Plugin writer, you ignore publications entirely.  You don't make it
> possible to do the wrong thing. You have something along the lines of a
> "PluginManagedDirectory" which core does not try to mess with.  If you want
> to implement rollback functionality, you do it through your own API where
> the side effects are more easily controlled and reasoned about.

+1 seems like the cleanest way to me

>
> I have doubts about whether Option 3 is viable - it seems like making it
> work reliably would be difficult.

I'd say option #1 and #3 are the same, #3 adding the complexity of
automating the rollback in Pulp,
option #2 and #4 are the same in the sense of Pulp staying away from
the incompatible workflow a content type has while providing a limited
functionality subset to the consumer. In addition, #4 allows for Pulp
service/host to provide both the Pulp-specific, limited functionality
as well as the incompatible, content-type specific workflows from a
"single" point. This might be a benefit to some folks.


Option #5: somehow make core Pulp (content versioning) compatible with

Re: [Pulp-dev] Core Commit Bit Process

[Milan Kovacik]

Brian,

thanks for the proposal!
I agree with the proposed process to gain the commit bit because of it
being based on the trust folks have to each other and the project and
because of its positive feedback loop effect on the trust building,
this process I believe will bring.
I'd like to suggest we adopt a process to relieve a commit bit too.

Thanks,
milan


On Mon, May 21, 2018 at 11:48 PM, Brian Bouterse  wrote:
> For core and it's related tools, we don't have a written process to describe
> giving the commit bit to a contributor. We've been wanting to agree on and
> document that process for a while, so I'm facilitating thread gathering
> ideas to inform the writing of a PUP.
>
> This starter email gives a brief history of what we've done and outlines a
> simple proposal to get us started. We can throw that proposal away in favor
> of any other idea.
>
> # History
>
> Historically if you were hired onto the Pulp team at Red Hat you received
> the commit bit day 0. In Oct 2017 we decided to stop doing that and instead
> document an open process. Engineers hired on the pulp time since Oct '17
> have not received commit bit. We have not yet documented an open process of
> which to give it to them or any other proven contributor.
>
> # Current State
>
> The current core devs as shown on github are: asmacdo, bizhang, bmbouter,
> daviddavis, dkliban, dalley, ipanova, jortel, pcreech, ttereshc
>
> # Scope of this discussion
>
> pulp/pulp, pulp/devel, and any repos for the Pulp3 Ansible installer. It
> applies to both Pulp2 and Pulp3. Plugins will do what they want.
>
> # Process Idea
>
> One process idea is to add a new core committer upon a vote with +1's
> received from all current core developers. The thinking is that all current
> core devs needs to be 100% comfortable for the new person to handle any
> issue in place of themselves.
>
> # Criteria
>
> Overall I believe someone who has demonstrated commitment and care towards
> the needs of all Pulp users and not only their own interests. Also they must
> have the experience to be trusted with major aspects of Pulp functionality.
>
> These requirements are somewhat vague by design. Any process with hard
> requirements will be gamed so I believe leaving it to the judgement of the
> existing devs is a safe approach. Anyone who specifically wants to get more
> involved should approach the core devs about mentorship. I think the right
> time will be obvious, and if there are doubts those can be expressed ahead
> of time or at vote time.
>
> # Code owners
>
> This commit bit vote could be for entire core repos, or it could be for a
> subsystem of Pulp enforced using github's "code owners" feature
> (https://blog.github.com/2017-07-06-introducing-code-owners/).
>
>
> ^ is starter content, please send ideas and discussion that will be
> incorporated into a first draft PEP at some point.
>
> -Brian
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp CLI MVP User Stories

[Milan Kovacik]

On Thu, May 17, 2018 at 8:14 PM, David Davis  wrote:

> Overall, I am hesitant to start with complex CLI commands. Instead I had
> hoped to start with basic, auto-generated CLI commands which we could then
> build off to add more complex commands in the future.
>

+1 ideally the CLI would provide some "mutable, high-level combinations" of
the basic CLI commands later on e.g by giving the user macros they could
reassemble on the CLI; considering the lisp-like Hylang[1][2][3] as an
example macro language:

  $ pulp-admin --eval '(create (repo-by-url  foo http://zoo.repo/ ))'
  {'repo': {'remote': {'name': 'zoo-remote', 'id': 1, 'url': '
http://zoo.repo'}, 'name': 'foo', 'id': 1}

could as well be specified by:

  $ pulp-admin --eval '(create (remote [[name hy_zoo] [url
http://zoo.repo/]]))'
  {'remote': {'name': 'Hy_zoo', 'url': 'http://zoo.repo', 'id': 2}}

  $ pulp-admin --eval '(create (repo [[name foo] [remote_id 2]]))'
  {'repo': {'remote': {'name': 'Hy_zoo', 'url': 'http://zoo.repo', 'id':
2}, 'name': 'foo', 'id': 2}}

or equally by:

  $ pulp-admin --eval '(create (repo [[name foo]] {'remote' {'name'
'Hy_zoo' 'url' 'http://zoo.repo' 'id' 3}}))'
   {'repo': {'remote': {'name': 'Hy_zoo', 'url': 'http://zoo.repo', 'id':
3}, 'name': 'foo', 'id': 3}}


or as well by:

  pulp-admin --eval '((->) (create (remote [[name Hy_zoo] [url
http://zoo.repo]]) (create (repo [name foo] ))'

so that the user is in control of the process rather than the converse ;)

--
milan

[1] http://docs.hylang.org/en/stable/index.html
[2] http://docs.hylang.org/en/stable/tutorial.html#macros
[3] http://docs.hylang.org/en/stable/tutorial.html#protips

PS: I'm not totally Hy right now ;)


> Questions/responses inline though.
>
> On Thu, May 17, 2018 at 11:52 AM, Dennis Kliban  wr
> ote:
>
>> The use cases we outlined earlier provide very little value over using
>> httpie to interact with the REST API directly.
>>
>
> I think this may be true for the first iteration of the CLI, but
> successive iterations will handle more complex use cases.
>

>
>> I'd like to propose 5 new use cases:
>>
>>- As a CLI user, I can create a repository, a remote, a publisher,
>>and a distribution with a single command.
>>- As a CLI user, I can create a repository version, a publication,
>>and update the distribution with a single command.
>>- As a CLI user, I can list remote types available on the Pulp server.
>>- As a CLI user, I can list publisher types available on the Pulp
>>server.
>>- As a CLI user, I can list all repositories available on the Pulp
>>server.
>>
>> How would users do things like update objects? Like what if I want to
> edit the URL I am syncing a repo from?
>
>
>>
>> The use cases proposed at the beginning on this thread require the user
>> to perform 4 steps before any content can be synced:
>>
>> 1) Create repository
>> 2) Create remote
>> 3) Create publisher
>> 4) Create distribution
>>
>> The goal for the CLI should be to reduce this to a single step. The CLI
>> will need to make some assumptions for the user: publisher name,
>> distribution name, auto publish, auto distribute, and maybe others.
>> However, this will allow the user to use a single command to create a
>> repository that's ready for sync/publish.
>>
>
> I think this is very brittle given the mutability of object names. Users
> couldn’t use both the CLI and REST API. They’d have to pick or choose or
> else run the risk of disconnecting objects.
>

  +1; we'd have to assume certain object name "shapes" such as
'repository-{name}'.format(name='foo')


>
> Also, how do we support things in the future like multiple remotes for a
> repo or shared remotes between repos?
>
>
 +1


>
>> Sync/Publish/Distribute workflow can also be 3 steps:
>>
>> 1) Create a new repository version
>> 2) Create a new publication
>> 3) Update distribution
>>
>> The goal here is to also reduce this to a single step.
>>
>> The other use cases are auxiliary.
>>
>> Questions? Thoughts? Ideas?
>>
>> -Dennis
>>
>>
>>
>>
>>
> A couple more questions:
>
> How would we handle things in the future that are plugin-specific like
> content upload/creation?
>
> Also, would this CLI use auto-generation at all? Or would it rely entirely
> on custom code? I assume the latter which means this would be similar to
> the Pulp 2 CLI in which every new API change required a change to the CLI
> code.
>
>
> David
>
> On Thu, May 17, 2018 at 11:52 AM, Dennis Kliban 
> wrote:
>
>> The use cases we outlined earlier provide very little value over using
>> httpie to interact with the REST API directly. I'd like to propose 5 new
>> use cases:
>>
>>- As a CLI user, I can create a repository, a remote, a publisher,
>>and a distribution with a single command.
>>- As a CLI user, I can create a repository version, a publication,
>>and update the distribution with a single command.
>>- As a CLI user, I can list 

Re: [Pulp-dev] Composed Repositories

[Milan Kovacik]

On Tue, May 15, 2018 at 4:48 PM, Jeff Ortel <jor...@redhat.com> wrote:
>
>
> On 05/15/2018 09:29 AM, Milan Kovacik wrote:
>>
>> Hi,
>>
>> On Tue, May 15, 2018 at 3:22 PM, Dennis Kliban <dkli...@redhat.com> wrote:
>>>
>>> On Mon, May 14, 2018 at 3:44 PM, Jeff Ortel <jor...@redhat.com> wrote:
>>>>
>>>> Let's brainstorm on something.
>>>>
>>>> Pulp needs to deal with remote repositories that are composed of
>>>> multiple
>>>> content types which may span the domain of a single plugin.  Here are a
>>>> few
>>>> examples.  Some Red Hat RPM repositories are composed of: RPMs, DRPMs, ,
>>>> ISOs and Kickstart Trees.  Some OSTree repositories are composed of
>>>> OSTrees
>>>> & Kickstart Trees. This raises a question:
>>>>
>>>> How can pulp3 best support syncing with remote repositories that are
>>>> composed of multiple (unrelated) content types in a way that doesn't
>>>> result
>>>> in plugins duplicating support for content types?
>>>>
>>>> Few approaches come to mind:
>>>>
>>>> 1. Multiple plugins (Remotes) participate in the sync flow to produce a
>>>> new repository version.
>>>> 2. Multiple plugins (Remotes) are sync'd successively each producing a
>>>> new
>>>> version of a repository.  Only the last version contains the fully
>>>> sync'd
>>>> composition.
>>>> 3. Plugins share code.
>>>> 4. Other?
>>>>
>>>>
>>>> Option #1: Sync would be orchestrated by core or the user so that
>>>> multiple
>>>> plugins (Remotes) participate in populating a new repository version.
>>>> For
>>>> example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote)
>>>> would both be sync'd against the same remote repository that is composed
>>>> of
>>>> both types.  The new repository version would be composed of the result
>>>> of
>>>> both plugin (Remote) syncs.  To support this, we'd need to provide a way
>>>> for
>>>> each plugin to operate seamlessly on the same (new) repository version.
>>>> Perhaps something internal to the RepositoryVersion.  The repository
>>>> version
>>>> would not be marked "complete" until the last plugin (Remote) sync has
>>>> succeeded.  More complicated than #2 but results in only creating truly
>>>> complete versions or nothing.  No idea how this would work with current
>>>> REST
>>>> API whereby plugins provide sync endpoints.
>>>>
>>> I like this approach because it allows the user to perform a single call
>>> to
>>> the REST API and specify multiple "sync methods" to use to create a
>>> single
>>> new repository version.
>>
>> Same here, esp. if the goal is an all-or-nothing behavior w/r the
>> mix-in remotes; i.e an atomic sync.
>> This has a benefit of a clear start and end of the sync procedure,
>> that the user might want to refer to.
>>
>>>> Option #2: Sync would be orchestrated by core or the user so that
>>>> multiple
>>>> plugins (Remotes) create successive repository versions.  For example:
>>>> the
>>>> RPM plugin (Remote) and the Kickstart Tree plugin (Remote) would both be
>>>> sync'd against the same remote repository that is a composition
>>>> including
>>>> both types.  The intermediate versions would be incomplete.  Only the
>>>> last
>>>> version contains the fully sync'd composition.  This approach can be
>>>> supported by core today :) but will produce incomplete repository
>>>> versions
>>>> that are marked complete=True.  This /seems/ undesirable, right?  This
>>>> may
>>>> not be a problem for distribution since I would imaging that only the
>>>> last
>>>> (fully composed) version would be published.  But what about other
>>>> usages of
>>>> the repository's "latest" version?
>>
>> I'm afraid I don't see use of a middle-version esp. in case of
>> failures; e.g ostree failed to sync while rpm managed and kickstart
>> managed too; is the sync OK as a whole? What to do with the versions
>> created? Should I merge the successes into one and retry the failure?
>> How many versions would this introduce?
>
>
> (option 2) The

Re: [Pulp-dev] Composed Repositories

[Milan Kovacik]

Hi,

On Tue, May 15, 2018 at 3:22 PM, Dennis Kliban  wrote:
> On Mon, May 14, 2018 at 3:44 PM, Jeff Ortel  wrote:
>>
>> Let's brainstorm on something.
>>
>> Pulp needs to deal with remote repositories that are composed of multiple
>> content types which may span the domain of a single plugin.  Here are a few
>> examples.  Some Red Hat RPM repositories are composed of: RPMs, DRPMs, ,
>> ISOs and Kickstart Trees.  Some OSTree repositories are composed of OSTrees
>> & Kickstart Trees. This raises a question:
>>
>> How can pulp3 best support syncing with remote repositories that are
>> composed of multiple (unrelated) content types in a way that doesn't result
>> in plugins duplicating support for content types?
>>
>> Few approaches come to mind:
>>
>> 1. Multiple plugins (Remotes) participate in the sync flow to produce a
>> new repository version.
>> 2. Multiple plugins (Remotes) are sync'd successively each producing a new
>> version of a repository.  Only the last version contains the fully sync'd
>> composition.
>> 3. Plugins share code.
>> 4. Other?
>>
>>
>> Option #1: Sync would be orchestrated by core or the user so that multiple
>> plugins (Remotes) participate in populating a new repository version.  For
>> example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote)
>> would both be sync'd against the same remote repository that is composed of
>> both types.  The new repository version would be composed of the result of
>> both plugin (Remote) syncs.  To support this, we'd need to provide a way for
>> each plugin to operate seamlessly on the same (new) repository version.
>> Perhaps something internal to the RepositoryVersion.  The repository version
>> would not be marked "complete" until the last plugin (Remote) sync has
>> succeeded.  More complicated than #2 but results in only creating truly
>> complete versions or nothing.  No idea how this would work with current REST
>> API whereby plugins provide sync endpoints.
>>
>
> I like this approach because it allows the user to perform a single call to
> the REST API and specify multiple "sync methods" to use to create a single
> new repository version.

Same here, esp. if the goal is an all-or-nothing behavior w/r the
mix-in remotes; i.e an atomic sync.
This has a benefit of a clear start and end of the sync procedure,
that the user might want to refer to.

>
>>
>> Option #2: Sync would be orchestrated by core or the user so that multiple
>> plugins (Remotes) create successive repository versions.  For example: the
>> RPM plugin (Remote) and the Kickstart Tree plugin (Remote) would both be
>> sync'd against the same remote repository that is a composition including
>> both types.  The intermediate versions would be incomplete.  Only the last
>> version contains the fully sync'd composition.  This approach can be
>> supported by core today :) but will produce incomplete repository versions
>> that are marked complete=True.  This /seems/ undesirable, right?  This may
>> not be a problem for distribution since I would imaging that only the last
>> (fully composed) version would be published.  But what about other usages of
>> the repository's "latest" version?

I'm afraid I don't see use of a middle-version esp. in case of
failures; e.g ostree failed to sync while rpm managed and kickstart
managed too; is the sync OK as a whole? What to do with the versions
created? Should I merge the successes into one and retry the failure?
How many versions would this introduce?

>>
>> Option #3: requires a plugin to be aware of specific repository
>> composition(s); other plugins and creates a code dependency between plugins.
>> For example, the RPM plugin could delegate ISOs to the File plugin and
>> Kickstart Trees to the KickStart Tree plugin.

Do you mean that the RPM plug-in would directly call into the File plug-in?
If that's the case then I don't like it much, would be a pain every
time a new plug-in would be introduced (O(len(plugin)^2) of updates)
or if the API of a plug-in changed (O(len(plugin)) updates).
Esp. keeping the plugin code aware of other plugin updates would be ugly.

>>
>> For all options, plugins (Remotes) need to limit sync to affect only those
>> content types within their domain.  For example, the RPM (Remote) sync
>> cannot add/remove ISO or KS Trees.
>>
>> I am an advocate of some from of options #1 or #2.  Combining plugins
>> (Remotes) as needed to deal with arbitrary combinations within remote
>> repositories seems very powerful; does not impose complexity on plugin
>> writers; and does not introduce code dependencies between plugins.
>>
>> Thoughts?
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

Cheers,
milan

[Pulp-dev] Request for feedback about rich dependencies implementation design

[Milan Kovacik]

Folks,

I'd like to ask Your feedback for $Subj, ideally on the
https://pulp.plan.io/issues/3528 issue.
Warning: long read ahead...

Thanks!
milan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] Issue #2619

[Milan Kovacik]

Folks,

I'd like to poll the channel for feedback about current implementation
and possible alternative(s) to it.
Issue #2619 TL;DR: report discrepancies between information kept in
Mongo and the state of (published) data kept on the disk[1]

Recent reviews are suggesting  to base the implementation on top of
relational data, kept in SQLite:
  - collect traits from a filesystem walk (checksums, sizes, link targets)
  - store these traits in separate tables
  - dump Mongo into unit, distributor (configuration) and repository tables
  - query the relational data to infer any discrepancies e.g broken
symlinks, wrong sizes or checksums
  - reuse the database for generating consequent reports

Current approach[2] TL;DR:
 - assemble a validation scenario based on CLI arguments e.g:
--check existence --check broken_rpm_symlinks --check size --check checksum
 - one by one, match the applicable content units from Mongo against
the validation scenario and filesystem traits
 - optionally skip checks that would fail for a unit e.g checksum
after invalid size
 - generate a report as a flat list of results per unit and
repository, in JSON format
 - perform consequent queries over the generated JSON report e.g
   jq '[.report[].repository] | unique' < report.json
   to get a list of affected repositories

There are some caveats with the current approach, such as:
 - in some cases, traits are first loaded into the memory from a
filesystem walk  e.g symlink targets
 - some repeated mongo queries are cached as well e.g distributors and
repositories
 - detecting broken symlinks still gives false negatives in corner cases

Cheers,
milan

[1] https://pulp.plan.io/issues/2619
[2] https://github.com/pulp/pulp_rpm/pull/1104
https://github.com/pulp/pulp/pull/3465

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Milan Kovacik]

I believe option 1 is a better fit as I think a new repository version
is a side-effect of a (plug-in specific) action on a repository.

Cheers,
milan

On Mon, Apr 16, 2018 at 10:31 PM, Dennis Kliban  wrote:
> On Fri, Apr 13, 2018 at 3:55 PM, Jeff Ortel  wrote:
>>
>>
>>
>> On 04/12/2018 04:49 PM, Dennis Kliban wrote:
>>
>> On Thu, Apr 12, 2018 at 2:49 PM, Jeff Ortel  wrote:
>>>
>>>
>>>
>>> On 04/11/2018 01:13 PM, Dennis Kliban wrote:
>>>
>>> On Tue, Apr 10, 2018 at 6:44 PM, Jeff Ortel  wrote:



 On 04/10/2018 04:15 PM, Dennis Kliban wrote:

 On Tue, Apr 10, 2018 at 2:04 PM, Brian Bouterse 
 wrote:
>
> These are good problem statements. I didn't understand all of the
> aspects of it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer
> to this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.
>

 I don't think all 3 are related problems. The motivation for grouping
 all together is that a subset of the action endpoints from problem 1 are
 used to create repository versions and Problem 3 is a problem with the
 repository version creation API.

>
>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald 
> wrote:
>>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>> Action endpoints are problematic.
>>
>> Example POST@/importers//sync/
>> They are non-RESTful and would make client code tightly coupled with
>> the server code.
>> These endpoints are inconsistent with the other parts of the REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe 
> "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not 
> RESTful"
> perhaps and have the self-consistency bullet removed?


 +1 to "Action endpoints are not RESTful"
 +1 to removing the self-consistency language

>>
>> DRF is not being used as intended for action endpoints so we have to
>> implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?
>
>>
>> We don't have a convention for where plug-in-specific, custom
>> repository version creation endpoints.
>>
>> example POST@/api/v3//docker/add/
>> needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed in
> the schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the 
> urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?
>


 Yes. I envision a CLI that can discover new plugin
 repository-version-creating functionality without having to install new
 client packages. Allowing plugin writers to add endpoints in arbitrary
 places for creating repository versions will make it impossible for the
 client to know what all the possible ways of creating a repository version
 are.

>>
>> For direct repository version creation, plugins are not involved.
>>
>> validation correctness problem: https://pulp.plan.io/issues/3541
>> example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
> plugin writers but not all.


 I think it affects all plugin writers. Even the File plugin needs to
 provide some validation when creating a repository version. Right now you
 can add a FileContent with the same relative path as another FileContent in
 the repository version. This should not be possible because it's not a 
 valid
 combination of FileContent units in the same repository version.


 Not necessarily.  Two files with the same relative path will have
 different digest (different content).  The assumption that they both cannot
 be in the same repository makes assumptions about how the repository is 
 used
 which I don't think is a 

Re: [Pulp-dev] Pulp 3 REST API Challenges

[Milan Kovacik]

Brian,

On Tue, Apr 10, 2018 at 8:04 PM, Brian Bouterse  wrote:
> These are good problem statements. I didn't understand all of the aspects of
> it, so I put some inline questions.
>
> My overall question is: are these related problems? To share my answer to
> this, I believe the first two problems are related and the third is
> separate. The classic divide and conquor approach we could use here is to
> confirm that the problems are unrelated and focus on resolving one of them
> first.

Addressing any of these issues separately influences the solution to
the rest of those therefore I believe them not to be independent.
For instance, keeping action endpoint importer/sync/ while
keeping POST@/repository//version/ while addressing the plug-in
involvement absence, results in a design where plug-in configuration
objects need to exist to be passed thru as parameters to those
endpoints and causing inconsistency with plug-in specific version
creation endpoints.
At the same time we are aware of a solution that addresses all three
issues at once by a design enhancement.
What would be the motivation to enumerate the issues separately?

>
>
> On Mon, Apr 9, 2018 at 3:18 PM, Austin Macdonald  wrote:
>>
>> Folks,
>>
>> Austin, Dennis, and Milan have identified the following issues with
>> current Pulp3 REST API design:
>>
>> Action endpoints are problematic.
>>
>> Example POST@/importers//sync/
>> They are non-RESTful and would make client code tightly coupled with the
>> server code.
>> These endpoints are inconsistent with the other parts of the REST API.
>
> Is self-consistency really a goal? I think it's a placeholder for
> consistency for REST since the "rest" of the API is RESTful. After reading
> parts of Roy Fielding's writeup of the definition of REST I believe "action
> endpoints are not RESTful" to be a true statement. Maybe "Action endpoints
> are problematic" should be replaced with "Action endpoints are not RESTful"
> perhaps and have the self-consistency bullet removed?

In my mind being RESTful implies consistency but I might be wrong.

>>
>> DRF is not being used as intended for action endpoints so we have to
>> implement extra code. (against the grain)
>
> I don't know much about this. Where is the extra code?

for instance, custom action endpoints need custom documentation
workarounds; see https://github.com/pulp/pulp_file/pull/60
TL;DR: working around POST@/type/ that doesn't create an instance of the type.

>
>>
>> We don't have a convention for where plug-in-specific, custom repository
>> version creation endpoints.
>>
>> example POST@/api/v3//docker/add/
>> needs to be discoverable through the schema
>
> What does discoverable via the schema ^ mean? Aren't all urls listed in the
> schema?
>
> I think of ^ problem somewhat differently. Yes all urls need to be
> discoverable (a REST property), but isn't it more of an issue that the urls
> which produce repo versions can't be identified distinctly from any other
> plugin-contributed url? To paraphrase this perspective: making a repo
> version is strewn about throughout the API in random places which is a bad
> user experience. Is that what is motivation url discovery?

The point here is to stress the absence of a good rule-of-a-thumb.
Currently POST@importers///sync/ and
POST@/repository//version/ create repository version.
Not being careful about a good design (by e.g preserving these core
endpoints), custom plug-in actions might end up being yet another
place where repository versions would be created.

>
>>
>> For direct repository version creation, plugins are not involved.
>>
>> validation correctness problem: https://pulp.plan.io/issues/3541
>> example: POST@/api/v3/repositories//versions/
>
> I agree with this problem statement. In terms of scope it affects some
> plugin writers but not all.

Yet a good design wouldn't stand in the way of plug-ins that won't
need the involvement by e.g not forcing a core endpoint to require
plug-in configuration to be passed thru.


Thanks,
milan

>
>>
>> We would like to get feedback on these issues being sound and worth
>> resolving before we resume particular solution discussion[1].
>>
>> Thanks,
>> Austin, Dennis, and Milan
>>
>> [1] https://www.redhat.com/archives/pulp-dev/2018-March/msg00066.html
>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Improving technical decision making

[Milan Kovacik]

Oh I'd forget that we actually don't really have a formal process to
recognize and retire active contributors yet;
how about the technical lead proposes both the recognition and
retirement anytime they find reason to do so, for the former
situation, with a pre-approval of  other active contributors, propose
folks publicly, for the latter situation, try reaching out to the
retiring contributor before going public to avoid frustration.
Folks of course would ideally announce their intention to retire, the
formal process would be conducted by the technical lead.
The insignia of an active contributor would be the commit bit on any
of the Pulp projects.
The first ever technical and community leads would be elected by folks
with the commit bit, the election would be organized by our current
community representatives.

Unless there are objections, I'd file a PUP to summarize these points.


Cheers,
milan

On Tue, Apr 3, 2018 at 6:02 PM, Milan Kovacik <mkova...@redhat.com> wrote:
> On Tue, Apr 3, 2018 at 3:47 PM, Austin Macdonald <aus...@redhat.com> wrote:
>> Interesting proposals Milan!
>>
>> I am forking Brian's email so that thread can focus on communication,
>> redmine, etc.
>
> Thanks, I guess it would best go hand-in-hand with the process update
> proposal for the Technical specifications/blue-prints PUP:
>
>>> I'd add that many a time, an e-mail based technical discussion gets
>>> messy and unfolds in multiple branches over multiple months.
>>> I'd like to propose we adopt a Technical Specification concept, living
>>> in a separate GitHub repo, similar to the PUP process.
>>> This would take advantage of our review process, preferably requiring
>>> multiple (core) reviewers acks before merging, allowing Redmine to be
>>> used for planning/tracking the (design) work.
>>> I think it's easier to manage the life-cycle of  a larger Technical
>>> Specification in a revision system document than an e-mail thread and
>>> a single Redmine issue.
>>> It also helps (feature) documentation and provides context.
>
>>
>> On Tue, Apr 3, 2018 at 8:13 AM, Milan Kovacik <mkova...@redhat.com> wrote:
>>>
>>> I'd also like to propose formal Project Technical Lead and a formal
>>> Project Community Lead roles to be able to decide in case of competing
>>> (technical) ideas or planning priorities.
>>> These would have to be time-boxed (half a year) and folks would elect
>>> the leader for a period based on leader's program, such as focus on
>>> particular goals for instance testing or refactoring.
>>> Any single person would be able to perform either the Community or the
>>> Technical Lead role in any given period but not both at the same time.
>>> The Community Lead role would take care for organizing the Technical
>>> Lead elections and vice versa, the Technical Lead would take care
>>> about organizing the Community Lead elections.
>>> The electorate would be the active contributors for both the roles.
>>> The candidates would be the active contributors too.
>>>
>>> This would open up the decision making process for anyone from the
>>> community, would encourage transparency, accountability and
>>> responsibility and would allow us to come to a decision on competing
>>> (technical) ideas or planning issues in case we'd got stuck.
>>>
>>> Cheers,
>>> milan
>>>
>>> On Mon, Apr 2, 2018 at 8:38 PM, Brian Bouterse <bbout...@redhat.com>
>>> wrote:
>>> > I agree the decision process for core itself needs discussion. For now,
>>> > I'm
>>> > only able to offer facilitating a convo that focuses on the
>>> > communication
>>> > aspects not the decision process. I would like to improve the
>>> > transparency
>>> > into the features that will and won't be in any given release for our
>>> > stakeholders. I hope we do discuss decision process as its own
>>> > discussion;
>>> > it's certainly deserving of a pup of its own.
>>> >
>>> > For the communication issues, soon I will share a basic outline of one
>>> > way
>>> > we could use Redmine for release planning. This would be a starter idea
>>> > towards a solution for us to modify together.
>>> >
>>> >
>>> >
>>> > On Mon, Apr 2, 2018 at 9:08 AM, Austin Macdonald <aus...@redhat.com>
>>> > wrote:
>>> >>
>>> >> I agree with the problems that Brian listed, but I hope we can focus on
>>> >> the decision making process itself in addition to how those decisions
>>> >> are
>>> >> communicated.
>>> >
>>> >
>>> >
>>> > ___
>>> > Pulp-dev mailing list
>>> > Pulp-dev@redhat.com
>>> > https://www.redhat.com/mailman/listinfo/pulp-dev
>>> >
>>
>>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

What still makes me feel most comfortable with the plug-in--centered
tasks proposal is that it doesn't try to conceal what actually happens
behind the scenes: a plug-in specific task code is fired all the time
a new repository version is created, that uses multiple (generic) core
Pulp objects to work on/with as mere URLs rather than verbose json
blobs passed around, and that no repository version gets created
without a plug-in involvement.


On Tue, Apr 3, 2018 at 6:01 PM, David Davis  wrote:
> I’ve had a chance to think about this some more this week and reread all the
> emails. I think that the solution of just adding a hook is best. Each
> solution seems to be imperfect and I think we still want users to interact
> with objects (remotes, repositories, etc). I’d say I’m +1 on adding a hook
> and -0 on the other proposals.

I wonder how the hook triggers would look like: what part of the core
would be responsible to select proper plug-in based on what criteria?
What if we later on need more action endpoints for instance repository
version "fork/clone"? Should we add more hooks then?
What if more fine-grained control of particular plug-in is required
for particular action endpoint? Should the plug-in hook just `if
endpoint == sync... elif endpoint == publish ... else...` it's way
thru?
How about custom plug-in action endpoint configuration? Should that be
passed-thru the generic core action endpoint? How that would be
auto-documented?
What if e.g pulp_docker doesn't need particular action endpoint, is it
cool to raise a 400? Isn't cleaner for both the plug-in writer and the
user not to have to expose such an endpoint in first place?
What about the plug-in custom async action endpoints that happen to
have to create a repository version?

Why workaround a design that doesn't seem to fit?

Cheers,
milan

>
>
> David
>
> On Mon, Apr 2, 2018 at 5:02 PM, Brian Bouterse  wrote:
>>
>> @asmacdo and I talked some and I wanted to share a few of my thoughts on
>> the plugin tasks problem statements.
>>
>> I agree with a problem statement for pulp_docker for example that it would
>> be good for a plugin writer to add custom validation when creating a new
>> repo/version. I thought this idea was stated already, but I'll retell what I
>> had thought we would do to add plugin writer validation. We would make an
>> optional plugin writer hook that validates the entire repository when the
>> RepositoryVersion is being made complete. Also this would also allow a
>> plugin to "disable" the repoversion endpoint by always raising an exception
>> here.
>>
>> The documentation convention is also a good outcome. It would recommend
>> that plugin writers put their views in a url namespaced by their plugin
>> name. I think we should do that.
>>
>> The other main problem I've read about with what we currently have is that
>> the actions urls (sync, publish, export, custom views) would be all in
>> different areas of the API urls. This is identified as a problem, but I
>> think this is ok. It allows us to explain each of those parts of pulp
>> separately, which makes the understanding of the API still pretty easy.
>>
>> I also believe another tertiary problem identified is that the 'sync',
>> 'publish', and 'export' words are not restful and it would be good to
>> resolve that somehow. A restful API would be an easier API to use because
>> REST clients already know how to interact with a resource. Action endpoints
>> kind of break this.
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 2, 2018 at 2:54 PM, Austin Macdonald 
>> wrote:
>>>
>>> Thanks for the classifications.
>>>
>>> On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:
>>>
 I think of these actions as 2 new types of resource in Pulp. Unlike
 Remotes(Importers currently) and Content, these resources are singletons
 that each plugin provides. Since users don't need to create new instances 
 of
 these resources it makes sense that they are represented by a View instead
 of a ViewSet. Though we don't even need to explain that to plugin writers.
 We just need to tell them that all operations that their plugin provides 
 for
 users need to be sublassed from ActionView.

 From the users point of view a GET on /api/v3/versionsactions/ to find a
 versionaction href is the same as performing a GET on /api/v3/remotes/ to
 find the href for a specific remote to sync from.

 Auto documentation would work perfectly.

>>>
>>> Its not that it would be "broken" or "wrong". I just think that users
>>> expect the documentation to include behavior and parameters-- ideally, there
>>> should be enough information to create a request. In this case however, the
>>> documentation just explains how to retrieve the necessary information. It
>>> works, but I don't think it is ideal that the user needs to make other GET
>>> requests to learn how to use an API 

Re: [Pulp-dev] Roadmap Challenges

[Milan Kovacik]

I'd also like to propose formal Project Technical Lead and a formal
Project Community Lead roles to be able to decide in case of competing
(technical) ideas or planning priorities.
These would have to be time-boxed (half a year) and folks would elect
the leader for a period based on leader's program, such as focus on
particular goals for instance testing or refactoring.
Any single person would be able to perform either the Community or the
Technical Lead role in any given period but not both at the same time.
The Community Lead role would take care for organizing the Technical
Lead elections and vice versa, the Technical Lead would take care
about organizing the Community Lead elections.
The electorate would be the active contributors for both the roles.
The candidates would be the active contributors too.

This would open up the decision making process for anyone from the
community, would encourage transparency, accountability and
responsibility and would allow us to come to a decision on competing
(technical) ideas or planning issues in case we'd got stuck.

Cheers,
milan

On Mon, Apr 2, 2018 at 8:38 PM, Brian Bouterse  wrote:
> I agree the decision process for core itself needs discussion. For now, I'm
> only able to offer facilitating a convo that focuses on the communication
> aspects not the decision process. I would like to improve the transparency
> into the features that will and won't be in any given release for our
> stakeholders. I hope we do discuss decision process as its own discussion;
> it's certainly deserving of a pup of its own.
>
> For the communication issues, soon I will share a basic outline of one way
> we could use Redmine for release planning. This would be a starter idea
> towards a solution for us to modify together.
>
>
>
> On Mon, Apr 2, 2018 at 9:08 AM, Austin Macdonald  wrote:
>>
>> I agree with the problems that Brian listed, but I hope we can focus on
>> the decision making process itself in addition to how those decisions are
>> communicated.
>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Roadmap Challenges

[Milan Kovacik]

Brian,

thanks for the proposal!
I've been considering a similar one for some time.

On Thu, Mar 29, 2018 at 9:13 PM, Brian Bouterse  wrote:
> I want to start a discussion around how Pulp does roadmap planning and some
> of our current challenges. This is the pre-discussion part of a future PUP.
> I want to collaborate on characterizing the problems first before we discuss
> solutions.
>
> # The surface problem statement
>
> It very difficult for external stakeholders to answer some simple questions
> about any given feature:
>
> * How would a user use this feature exactly?
> * Is it completed? If not, how much is left to do?
> * Which release is this going in?
> * Has this feature been fully planned and accepted as a committed to piece
> of work?
>
> # deeper problems
>
> I believe there are two deeper problems contributing to the problem above.
>
> 1. Any given feature is typically spread across multiple Redmine tickets.
> There may be the first implementation, followup changes, adjustments,
> bugfixes, reworks, docs followups, etc. This makes it practically hard to
> have the information necessary to answer the first 3 questions ^.
>
> 2. Devs of core or a plugin have no clear way to clearly signal that a
> feature has been fully planned and is committed to. The 'sprint' field has
> been used heretofore, but the recent feedback suggests that mechanism may
> not be the best way to indicate that work has been fully planned and
> accepted. We need a clear way to answer the last question ^.
>
> Do you agree or disagree with these problem statements? Do you have anything
> to add about the problem statements?
>

I'd add that many a time, an e-mail based technical discussion gets
messy and unfolds in multiple branches over multiple months.
I'd like to propose we adopt a Technical Specification concept, living
in a separate GitHub repo, similar to the PUP process.
This would take advantage of our review process, preferably requiring
multiple (core) reviewers acks before merging, allowing Redmine to be
used for planning/tracking the (design) work.
I think it's easier to manage the life-cycle of  a larger Technical
Specification in a revision system document than an e-mail thread and
a single Redmine issue.
It also helps (feature) documentation and provides context.


> Thanks!
> Brian
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

Cheers,
milan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

Folks,

I'd like to summarize where we've gotten so far. Our problem statement
is threefold:

1) Content operations have to be routed thru particular plug-ins i.e
no repository version can be created without a plug-in involvement, to
avoid repository content consistency/correctness issues
2) API specialization i.e keeping core concepts above plug-in specific
concepts; e.g CRUD@/v3/repositories/ v.s CRUD@docker/adds/
3) API consistency i.e preferably where I create it is where I manage
it; e.g. w/r/t asynchronous action endpoints creating Task resources
vs a Task management endpoint

Since there is a significant intersection between the asynchronous
action endpoints and endpoints manipulating repository versions
asynchronously, me and Austin propose to convert the asynchronous
action endpoints to Task management endpoints, in full control of
plug-ins, in particular:

Asynchronous Repository Version Management Endpoints:

  Repository synchronisation and publication and addition(deletion) of
content into(from) a repository version would all be treated as Task
objects primarily, residing in the plug-in namespace:
  POST@/v3/tasks//[syncs, publishes, additions, deletions]/
  because all these operations share similar call signature, every one
of these actions:

  * creates a Task object
  * the task, while executing, creates a new repository version and
locks similar resources
  * has to be controlled by a particular plug-in i.e may require
additional, plug-in specific settings
  * requires either a repository URI or repository version URI to operate on
  * requires generic action configuration e.g almost every sync
requires (at least one) importer

 As a result:
   * the asynchronous content management Task is now the nexus where
the configuration (repository, plug-in, origin) meets the action
   * these complex tasks that actually do the heavy lifting have a
clear management endpoint
   * this change effectively renders the
@/v3/repositories//versions/<#nr>/ endpoint read-only and
documents the intention that no repository version is ever created
without an actual plug-in involvement
   * a plug-in can expose custom asynchronous content endpoints, that
manipulate repository versions, aligned with the more generic actions:
   POST@/v3/tasks//[,...]/
   * the design aligns better with the DRF allowing (nested)
serializers to control (and document) the endpoints in the direction
from the more generic to the more specific implementation

Asynchronous Core Object Management Endpoints:

  Additionally, for Asynchronous Core object management endpoints, we
suggest to move Remote/Importer, Publisher, Exporter, Repository
operations under a new core task namespace:
   POST@/v3/tasks/core/[importers, publishers,
repositories]/[deletes, updates]/
  to be consistent with the asynchronous content actions. This however
seems to conflict with 3).

Caveats

Both the current and the proposed design don't conceptually prevent a
repository to have content provided by multiple plug-ins.
Folks don't like Pulp becoming a task management system.

Alternatives

  Dennis proposes to achieve similar result by adding plug-in content
model hooks:
  "add_to_repository_version" and "remove_from_repository_version"
  This approach however doesn't seem to solve plug-ins requiring
multiple or no importers and leaves multiple spots in the API, where
repository versions can be created (failing 3); custom plug-in
asynchronous content management endpoints and importers/remotes sync
vs POST@v3/repository/UUID/versions/<#nr>/

  David proposes to namespace all objects under plug-in specific
endpoints. This solves the repository--plug-in purity caveat, allows
great plug-in flexibility and prevents endpoint name collisions.
 This approach may allow the plug-ins UX to drift from each other
though, and solves neither the multiple/no importers case, nor the
scattering of repository versions creation endpoints and fails 2)&3).

I believe this Master/Detail Tasks design to be a reasonable
compromise on the requirements in 1) 2) 3).

Cheers,
milan

On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban  wrote:
> On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova  wrote:
>>
>> I do not think that it's a valid argument to ban the proposal just because
>> the proposal will bring changes to the existing plugins. Because:
>>
>> 1) i think it would be fair to think of other plugins and find a solution
>> all plugins will be happy with --> so we are back to the generic correctness
>> problem
>> 2) we are not GA, not even Beta, we are free and eligible to make changes,
>> no promises made yet.
>> 3) if we bring changes *this* exactly is the time to do because we have
>> just 2 plugins working with basic functionality :)
>>
>> W/r to the "Pulp turning into a task running system" ..and yet that's a
>> system based on distributed task system
>
>
> Pulp may rely on a tasking system to get work done, but we don't want 

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Tue, Mar 27, 2018 at 12:00 AM, Brian Bouterse  wrote:
> I also want to look at how this will affect our user experience. With this
> change, I believe plugin tasks are put at the center of the user experience
> and the repositories, content, and versions are moved to the background. The
> user would think about what they can do foremost as a series of tasks. This
> is great in terms of organization, but I'm concerned its stepping away from
> what makes pulp special. I think Pulp is special because of its focus on
> content management (repos, version, uploads, sync, promotion, copy/delete,
> publish, export).
Fwiw you just enumerated 7 verbs vs 2 nouns ;)

> Having the user think about Pulp as a task-running system
> takes Pulp a step closer to a software like RunDeck
> (https://www.rundeck.com/open-source).
>
> What I like about the existing user experience (and API) is that it all
> centers around Pulp's value proposition (importers, syncs, associations,
> etc).
The Pulp focus isn't repo/version/content/ objects CRUD.
The value added are those uploads, syncs, promotions, copies,
publishes and exports.

If we don't like 'tasks' we can have a poll about a better name of the
endpoint; 'actions' anyone? Or perhaps 'deeds'?

>
> Does this spark any thoughts or ideas from others?
>
>
> On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:
>>
>> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
>> wrote:
>>>
>>>
>>>
>>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>>> wrote:

 This proposal does not make the plugin writer's job any easier. This
 simply changes where the plugin writer is providing validation logic, in a
 serializer vs. in a view.
>>>
>>>
>>> It does make the plugin writer's job easier *because* it changes how the
>>> validation is provided, Plugin writers already have to understand how to
>>> create models/serializers (and it is very simple) for all the other objects.
>>> Seriallizer validation is much simpler than viewset validation because
>>> serializers are literally designed to do this.
>>
>>
>> If we don't change anything, a plugin writer is responsible for creating a
>> task that creates a Repository Version and a view which will perform
>> validation and then dispatches a task. If we do make the proposed change,
>> the plugin writer will still have to write the task code and then also a
>> serializer that performs validation. The proposed change only removes the
>> responsibility of dispatching a task which is 1 line of code.
>>
>>>
>>>


 The other problem I see is that complexity for users is increased.
 Instead of having 1 resource for tracking task progress, there would be an
 infinite number of resources for tracking progress of tasks.
>>>
>>>
>>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>>> the task they are looking for (v3/tasks/file/syncs/)
>>
>>
>> My understanding of the master/detail pattern we are using for our other
>> resources is that each task's URI will look something like this:
>> /api/v3/tasks/file/syncs//
>>
>> If the above is true, that means the client has to be aware of
>> '/api/v3/tasks/file/syncs//' resource. The user would need to know how
>> to interpret the response from each task type. There could be an infinite
>> number of these tasks with many different arguments. So writing client code
>> handling all those responses could become a nightmare. Right now the user
>> just needs to know how to interpret the response from '/api/v3/tasks/'
>> for all tasks.
>>
>>>
>>>

 I don't see the value added by the proposed change.
>>>
>>>
>>> Value #1: We have to do something to address the problem that
>>> adding/removing content to a repository without plugin input is incorrect.
>>> This proposal is one possibility, but it isn't valid to compare the value
>>> against doing nothing. Instead, if you don't like this option, we need to
>>> compare it to other proposals for how to involve plugins in tasks.
>>>
>>
>> We don't have a problem. I agree that a user needs to know a whole lot of
>> information to correctly create a working repository version for a complex
>> content type such as a Docker manifest. Without all this information on
>> hand, the user could easily create a broken repository version. However, a
>> rich client could solve that problem. If the plugin writer wants to support
>> simpler clients, she can provide an additional URL for handling the
>> validation and additional logic for creating a repository version. We should
>> probably have a recommended convention for plugin authors to use when adding
>> additional URLs.
>>
>>>
>>> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
>>> shouldn't need to understand the 

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Tue, Mar 27, 2018 at 12:02 AM, Austin Macdonald  wrote:
>
>
> On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:
>>
>> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
>> wrote:
>>>
>>>
>>>
>>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>>> wrote:

 This proposal does not make the plugin writer's job any easier. This
 simply changes where the plugin writer is providing validation logic, in a
 serializer vs. in a view.
>>>
>>>
>>> It does make the plugin writer's job easier *because* it changes how the
>>> validation is provided, Plugin writers already have to understand how to
>>> create models/serializers (and it is very simple) for all the other objects.
>>> Seriallizer validation is much simpler than viewset validation because
>>> serializers are literally designed to do this.
>>
>>
>> If we don't change anything, a plugin writer is responsible for creating a
>> task that creates a Repository Version and a view which will perform
>> validation and then dispatches a task. If we do make the proposed change,
>> the plugin writer will still have to write the task code and then also a
>> serializer that performs validation. The proposed change only removes the
>> responsibility of dispatching a task which is 1 line of code.
>
>
> The difference is not in the number of lines, it is in the complexity.
> Validation with serializers is simpler, and more similar to other plugin
> code.
>
> New Way:
> https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L15-L21
> Old
> way:https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L13-L30
   oh, and let's not forget the boilerplate needed to make the old way
autodoc working for the custom action endpoints:
https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L31,#L61
   I dare to say majority of the sync objects (serializer) boilerplate
will be generic enough to be shared between the plugins; there's not
much one can do about a sync request but creating it...

>
> That "1 line of code" puts the plugin writer in contact with a very complex
> part of the pulpcore internals. `apply_async_with_reservation`:
> https://github.com/pulp/pulp/blob/e8f804dc6595732dad38ec478877d919d7d97063/pulpcore/pulpcore/tasking/tasks.py#L180
>

 The other problem I see is that complexity for users is increased.
 Instead of having 1 resource for tracking task progress, there would be an
 infinite number of resources for tracking progress of tasks.
>>>
>>>
>>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>>> the task they are looking for (v3/tasks/file/syncs/)
>>
>>
>> My understanding of the master/detail pattern we are using for our other
>> resources is that each task's URI will look something like this:
>> /api/v3/tasks/file/syncs//
>>
>> If the above is true, that means the client has to be aware of
>> '/api/v3/tasks/file/syncs//' resource. The user would need to know how
>> to interpret the response from each task type. There could be an infinite
>> number of these tasks with many different arguments. So writing client code
>> handling all those responses could become a nightmare. Right now the user
>> just needs to know how to interpret the response from '/api/v3/tasks/'
>> for all tasks.
>
>
> I might be missing your point on this one. If you are talking about polling
> tasks, the href of a new Task is returned when the Task is created.The
> client should use the href that is returned, never string formatting urls.
> Even so, the Tasks in this proposal will be part of the API schema, so
> clients (and docs users) know where the endpoints are and what the arguments
> should be. This would actually be a big advantage over the way things are
> now. AFAIK, v3/importers/1234/sync/ is invisible to clients, so we replace
> one invisible endpoint with one visible endpoint. No complexity added.
>
> If the client doesn't know how to interpret extra fields (the Task
> parameters), they don't need to. All tasks share the same base fields, and
> those are the fields that need to be interpreted when the client is polling
> a task.
+1 all tasks will share common ancestor and path in the url; it's very
easy to explore and follow, see also the autodoc screen cast

>
>>
>> Value #1: We have to do something to address the problem that
>> adding/removing content to a repository without plugin input is incorrect.
>> This proposal is one possibility, but it isn't valid to compare the value
>> against doing nothing. Instead, if you don't like this option, we need to
>> compare it to other proposals for how to involve plugins in tasks.
>>
>>
>> We don't have a problem. I agree that 

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Mon, Mar 26, 2018 at 9:38 PM, Austin Macdonald  wrote:
>
>
> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban  wrote:
>>
>> This proposal does not make the plugin writer's job any easier. This
>> simply changes where the plugin writer is providing validation logic, in a
>> serializer vs. in a view.
>
>
> It does make the plugin writer's job easier *because* it changes how the
> validation is provided, Plugin writers already have to understand how to
> create models/serializers (and it is very simple) for all the other objects.
> Seriallizer validation is much simpler than viewset validation because
> serializers are literally designed to do this.
>
>>
>>
>> The other problem I see is that complexity for users is increased. Instead
>> of having 1 resource for tracking task progress, there would be an infinite
>> number of resources for tracking progress of tasks.
>
>
> In the proposal, Tasks are Master/Detail. The user doesn't have to "track"
> it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
> tasks will either be the same (v3/tasks/) or easier if they know the task
> they are looking for (v3/tasks/file/syncs/)
>
>>
>> I don't see the value added by the proposed change.
>
>
> Value #1: We have to do something to address the problem that
> adding/removing content to a repository without plugin input is incorrect.
> This proposal is one possibility, but it isn't valid to compare the value
> against doing nothing. Instead, if you don't like this option, we need to
> compare it to other proposals for how to involve plugins in tasks.
>
> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
> shouldn't need to understand the complexities of our tasking system.
> Pulpcore can handle task dispatching generally, plugin writers just set
> simple class attributes instead.
>
> Value #3: It removes "action" endpoints and replaces them with RESTful
> "creation of task" POSTs.
>
> Value #4: It creates a convention for endpoints. Right now we have
>
>
> v3/importers/file/1234/sync/
> v3/publishers/python/1234/publish/
>
> There is no logical place for add/remove to go because add/remove will not
> involve an importer. It will be irritating to the users if we don't
> encourage some consistency between plugins for the simplest Pulp
> functionality.
>
>
> Value #5: Task history becomes much more useful. Currently, the only data on
> a task that connects it to user-facing objects is "created_resource". This
> proposal will allow users to filter tasks by parameters. For example, they
> can filter sync tasks by "repository" or "importer". They can also use the
> detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
> based on the plugin or action.
>
> Value #6: Since the parameters of Task are saved, it will be possible to
> repeat tasks.

btw A moving picture of how it's going to look like on the docs side
of things: https://youtu.be/E7zC6SElm4g ;)

--
milan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Port Pulp3 to use RQ

[Milan Kovacik]

+1 I like RQ and I like http://python-rq.org/docs/testing/ esp.
there's Fakeredis ;)


--
milan


On Thu, Mar 22, 2018 at 6:58 PM, Brian Bouterse  wrote:
> Thanks for all the discussion both on list and on irc. After more
> investigation, it sounds like there are no feature gaps, but we will need to
> incorporate this workaround to cancel a task that is already running.
>
> The feedback I've heard on the idea is that it's valuable and looks
> feasible, but we won't really know until we prototype it a bit. Based on the
> technical outline in the previous email, I believe it can be prototyped in a
> day or two. I plan to do this soon, once I contribute to a few other
> required-for-beta planning items first. I'll post my PR to see what other
> think of the change, probably next week.
>
>
>
> On Wed, Mar 21, 2018 at 6:41 PM, Daniel Alley  wrote:
>>
>> I meant in the sense that, what is the aftermath when it comes back
>> online, and is it screwed up in ways that cause side effects.
>>
>> On Wed, Mar 21, 2018 at 5:02 PM, Jeremy Audet  wrote:
>>>
>>> > RQ does not support revoking tasks.  If you send the worker a SIGINT,
>>> > it will finish the task and then stop processing new ones.  If you send 
>>> > the
>>> > worker SIGKILL, it will stop immediately, but I don't think it gracefully
>>> > handles this circumstance.
>>>
>>> Nothing handles SIGKILL gracefully. Processes can't catch that signal.
>>> `kill -9 $pid` sends SIGKILL.
>>>
>>> If one is looking for a way to gracefully, immediately kill an RQ
>>> worker, then SIGTERM may do the trick. Anecdotally, many processes
>>> handle this signal in a hurried fashion. Semantically, this is
>>> appropriate: SIGINT is the "terminal interrupt" signal (Ctrl+c sends
>>> SIGINT), whereas SIGTERM is the "termination signal."
>>
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik <mkova...@redhat.com> wrote:
>
>
>
> On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald <aus...@redhat.com> wrote:
>>
>> I spoke with daviddavis about this and I would like to narrow the scope a 
>> bit. This discussion should be limited to endpoints that deploy tasks. The 
>> possibility for collisions that David pointed out regarding 
>> v3/content// is real, but should be discussed separately because 
>> Content objects should follow the pattern of the other plugin defined 
>> objects. All of them are master/detail so the namespacing 
>> v3/plugin//content/ would go deeply against the grain.
>>
>>
>> On Mar 15, 2018 10:53, "David Davis" <davidda...@redhat.com> wrote:
>>>
>>> I have an amendment to this proposal. Instead of namespacing just the 
>>> plugin task routes, I’d propose we namespace all plugin routes. Thus all 
>>> plugin routes would be namespaced under something like 
>>> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets 
>>> moved to “/api/v3/plugin/file/content/” and so on.
>>>
>>>
>>> For task routes, plugins could still route to a pulpcore viewset/action:
>>>
>>> User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]
>>>
>>> Routes to: POST /api/v3/tasks plugin=file task=create_repository_version 
>>> add_content=[…]
>>
>>
>> AFAIK, we don't have a good mechanism for validating and re-routing 
>> endpoints quite like this. Instead, a view or viewset would be created, 
>> validation performed, and then a task is deployed. The task itself could be 
>> a vanilla task from pulpcore though. Another issue is that a POST to 
>> v3/plugin/file/repository_version/ implies that the resultant 
>> repository_version is typed in some way. The created resource href would 
>> still take the form v3/repositories/1234/versions/2/, so I think this is a 
>> little misleading.
>
>
> +1 to generic Pulp concepts/objects staying outside of a plug-in 
> namespace/path
>
>>
>>
>> For the other endpoint:
>>  /api/v3/tasks plugin=file task=create_repository_version add_content=[…]
>>
>> There is a still the problem is that POST requests should not contain 
>> parameters that influence the allowable parameter set. For instance, 
>> plugin=python might require an additional parameter that is not allowed for 
>> the file plugin. In particular, this affects the auto-documented REST API.
>>
>>
>> However, the concept in general could work. I've adapted David's suggestion 
>> and will present it side by side with the original idea. Note, this only 
>> applies to "actions", which are sync, publish, add/remove, and 
>> plugin-specific actions (including rich-dependency adding).
>>
>> The difference between the two ideas is based in semantics. Both would 
>> deploy the same task functions.
>>
>> "Action Based"
>> POST v3/plugin/file/sync/
>> Note that "sync" is singular. This is a file-plugin action to sync using the 
>> body parameters.
>
>
> - 1 as this might imply mixing  and  actions while 
> all the time, asynchronous actions, such as the sync call, always create a 
> task object; the user would be creating the task objects at one endpoint but 
> deleting, filtering, tracking those at another.
>
>>
>>
>> "Task Object Based"
>> POST v3/tasks/file/syncs/
>> Note that "syncs" is plural. This is a POST that creates a file-sync task 
>> object,.

Btw an interesting consequence of tasks not being objects in a file
importer and publisher are these two issues:
 - The parameters in the API schema for the sync and publish endpoints
are incorrect [1]
 - Not able to access documentation endpoint [2]

In the former issue, the DRF isn't able to automatically figure out
schema for e.g the `sync` endpoint of the file plug-in importer.
The reason for it is even though the `sync` verb is attached to an
importer object, the parameters passed in aren't supposed to be
importer details so they shouldn't use the importer serializer to get
e.g auto-documented.
Moreover, the result of a call to this endpoint is supposed to be a
task (json object), schema and serialization of which has to be
repeated for the `sync` endpoint to some extent.
Last but not least, it isn't clear whether the `sync` verb should
belong to an importer or a repo, or something else as the call
"responsibility" is spread somewhat between both and none.

The latter issue is a consequence of the lack of support in DRF for
what the `sync` ver

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald  wrote:

> I spoke with daviddavis about this and I would like to narrow the scope a
> bit. This discussion should be limited to endpoints that deploy tasks. The
> possibility for collisions that David pointed out regarding
> v3/content// is real, but should be discussed separately because
> Content objects should follow the pattern of the other plugin defined
> objects. All of them are master/detail so the namespacing
> v3/plugin//content/ would go deeply against the grain.
>
>
> On Mar 15, 2018 10:53, "David Davis"  wrote:
>
> I have an amendment to this proposal. Instead of namespacing just the
> plugin task routes, I’d propose we namespace all plugin routes. Thus all
> plugin routes would be namespaced under something like
> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
> moved to “/api/v3/plugin/file/content/” and so on.
>
>
> For task routes, plugins could still route to a pulpcore viewset/action:
>
> User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]
>
> Routes to: POST /api/v3/tasks plugin=file task=create_repository_version
> add_content=[…]
>
>
> AFAIK, we don't have a good mechanism for validating and re-routing
> endpoints quite like this. Instead, a view or viewset would be created,
> validation performed, and then a task is deployed. The task itself could be
> a vanilla task from pulpcore though. Another issue is that a POST to
> v3/plugin/file/repository_version/ implies that the resultant
> repository_version is typed in some way. The created resource href would
> still take the form v3/repositories/1234/versions/2/, so I think this is
> a little misleading.
>

+1 to generic Pulp concepts/objects staying outside of a plug-in
namespace/path


>
> For the other endpoint:
>  /api/v3/tasks plugin=file task=create_repository_version add_content=[…]
>
> There is a still the problem is that POST requests should not contain
> parameters that influence the allowable parameter set. For instance,
> plugin=python might require an additional parameter that is not allowed for
> the file plugin. In particular, this affects the auto-documented REST API.
>

> However, the concept in general could work. I've adapted David's
> suggestion and will present it side by side with the original idea. Note,
> this only applies to "actions", which are sync, publish, add/remove, and
> plugin-specific actions (including rich-dependency adding).
>
> The difference between the two ideas is based in semantics. Both would
> deploy the same task functions.
>
> "Action Based"
> POST v3/plugin/file/sync/
> Note that "sync" is singular. This is a file-plugin action to sync using
> the body parameters.
>

- 1 as this might imply mixing  and  actions
while all the time, asynchronous actions, such as the sync call, always
create a task object; the user would be creating the task objects at one
endpoint but deleting, filtering, tracking those at another.


>
> "Task Object Based"
> POST v3/tasks/file/syncs/
> Note that "syncs" is plural. This is a POST that creates a file-sync task
> object,.
>

+ 1 for typed objects

# adding content to a repo version (already mentioned by Austin but I
like it)
POST@v3/tasks/docker/content_additions/

# synchronize the Zoo repo
pulp-admin create tasks rpm syncs --repo-id zoo --importer-id
zoo-importer --sync-config-override '{"ssl": "false"}' --format=json | tee
/tmp/inttest01_sync_task.json | pulp-admin track tasks --format json -

# get all docker content additions tasks since yesterday
pulp-admin list tasks docker content_additions --format=csv
--fields=id,worker_id --filter=started_at__gt=

# get all tasks since yesterday
pulp-admin list tasks  --filter=started_at__gt= --format=yaml


> Discussion:
> Another consideration is "Where will a Live API live?". A benefit of
> David's proposal is that we would be providing plugins with a namespaced
> area to do whatever they need, from live api to actions. We probably would
> want to create this namespace for things like live apis even if we go in
> the tasks/file/syncs/ direction. The only downside of this part of David's
> proposal is that our endpoints will still be "action" verbs instead of
> nouns, and I am ok with that.
>

I'd suggest live API actions that create task object subclasses (generic
Pulp concept) reside in namespaced tasks path; this would still allow the
user to apply generic concepts
POST@/v3/tasks/file/frobnications/

pulp-admin create tasks file frobications --foo=bar --format json |
pulp-admin track tasks --format json --

Other live API endpoints (not creating task objects) might be kept under
namespaces
POST@v3/plugins/file/shake/<{level: strong}

pulp-admin plugins file shake --level=strong


> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
Cheers,

Re: [Pulp-dev] 2.15.3 dev freeze -- 22:00 UTC Tuesday, March 6th

[Milan Kovacik]

On Tue, Mar 6, 2018 at 11:30 PM, Brian Bouterse  wrote:

> Per the release schedule, the code for 2.15.3 is now frozen. There are a
> total of 8 issues spread across core, rpm, and docker. This is ready for
> release engineering. The current beta date for 2.15.3 is March 13th.
>
> https://pulp.plan.io/issues?query_id=59
>
> @milan, these two issues [0][1] are shown as stories so they are not
> included in 2.15.3. They should be included in 2.16 though since they are
> already merged. Will that work?
>

Both stories turned out to be documentation issues only (lack of current
state of affairs description) and in both the cases the patchs are just
straightforward (small additions) docs updates so I'd say should be cool to
cherry-pick into 2.15.3 anyway.  Unsure though how to make that formally
appealing; a comment in the issue/story?

Cheers,
milan


>
> [0]: https://pulp.plan.io/issues/3353
> [1]: https://pulp.plan.io/issues/3339
>
> Thanks,
> Brian
>
>
> On Mon, Mar 5, 2018 at 11:18 AM, Brian Bouterse 
> wrote:
>
>> Any Pulp2 core or plugin code that you want included in the 2.15.3
>> release must be:
>>
>> a) merged to master by 22:00 UTC, March 6th
>> b) be associated with a bugfix issue. stories, refactors, and tasks are
>> not included in z-stream releases.
>>
>> If you want/need to adjust this date, please reply reply on-list.
>>
>> Thanks!
>> Brian
>>
>
>
> On Mon, Mar 5, 2018 at 11:18 AM, Brian Bouterse 
> wrote:
>
>> Any Pulp2 core or plugin code that you want included in the 2.15.3
>> release must be:
>>
>> a) merged to master by 22:00 UTC, March 6th
>> b) be associated with a bugfix issue. stories, refactors, and tasks are
>> not included in z-stream releases.
>>
>> If you want/need to adjust this date, please reply reply on-list.
>>
>> Thanks!
>> Brian
>>
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin Writer's Coding Workshop Feedback

[Milan Kovacik]

>
> - a database picture would be helpful to show off the data layer objects,
> foreign keys, and attributes.
>
>
> Yes!  We really need to publish an ER diagram.  I'm overdue on an action
> item to produce one.
>

https://pulp.plan.io/issues/3369
​ totally missed this one​


--
​milan​
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin Writer's Coding Workshop Feedback

[Milan Kovacik]

Brian,

thanks for the summary!
I took the liberty to go thru this list and file Redmine issues where I
found the description to be straightforward; please see links inline.
My intention is to be able to track and triage these so that the invaluable
plug-in writer feedback isn't lost.

Cheers,
milan

On Mon, Feb 12, 2018 at 10:57 PM, Brian Bouterse 
wrote:

> At the Foreman Construction day [0] last Wednesday, we had our first code
> focused plugin writer's workshop. About 6 people were actively engaged as
> we talked through the plugin API, example code, and then tried to install
> Pulp3. All of this happened over about 4-5 hours. In contrast to the
> devconf workshop which was planning focused, this was a "let's look at and
> write some code together" workshop. Two attendees came to both, and they
> got all the way to calling their own sync code.
>
> We got a lot of feedback, which I will try to group into some areas. (my
> feedback in parens)
>
> [installation issues]
> - the pypi install commands are missing the migrations and they produce
> broken installations
>

https://pulp.plan.io/issues/3362


> - the vagrantcloud boxes couldn't have a plugin installed on them :(
> - the dev environments worked great but we didn't recommend them until we
> realized all of these other methods were broken
> - we assume the user 'pulp' in a lot of places, e.g. systemd file,
> ansible, etc
> - assumptions about Fedora both in ansible, but also the copy/paste
> commands
> - some users who copied and pasted commands didn't realize they weren't
> for their OS
>
> [desire for simpler things]
> - there is a strong desire to use sqlite as the default db not postgresql
>

https://pulp.plan.io/issues/3359


> - desire to not install a message bus. (I think this is unavoidable)
>
>
https://pulp.plan.io/issues/3361


> [need examples]
> - pulp_file is our example, but it's laid out into different functions and
> classes. People were confused by this because they thought the classes and
> function names are meaningful when they aren't. For example we were asked
> "what is a synchronizer" https://github.com/pulp/pulp_
> file/blob/master/pulp_file/app/tasks.py#L139
> - pulp_file doesn't provide a good example because changesets do
> everything for you. (The main pulp_file should be a simple, direct example
> of the objects they have to save).
> - people found pulp_example via github and thought "oh here is what I
> needed to find!" only to base their code on outdated code (we need to
> delete pulp_example)
> - a database picture would be helpful to show off the data layer objects,
> foreign keys, and attributes.
>
> [specific things]
> - 'id' on the inherited content unit conflicted with a content unit which
> also uses 'id'.
>

https://pulp.plan.io/issues/3365


> - qpid vs rabbitmq defaults confusion. The settings.yaml says we default
> to qpid so they installed qpid, but really in settings.py it's rabbitmq.
> (this is a 1 line fix)
>
>
https://pulp.plan.io/issues/3366


> In terms of the installation challenges, we should consider consolidating
> onto a single installation method of pip with virtualenv. Of all the
> methods we offer [1] that is the one everyone could use and no one minded.
> We could remove the other options from the install page so that for for now
> (pre-GA) everyone is doing the same, simple thing. I think we should
> consolidate our effort and not focus on end-user installations as the main
> thing right now.**
>
> I also think we should do these things:
>
> * switch pulp to use sqlite3 by default. An ansible installer can both
> install postgres and configure it, later.
> * rewrite pulp_file to be a really really simple example
>

https://pulp.plan.io/issues/3367


> * delete pulp_example
>
>
https://pulp.plan.io/issues/3364


> Please send ideas, questions, or any kind of feedback.
>
> [0]: http://cfgmgmtcamp.eu/fringe.html#foreman
> [1]: https://docs.pulpproject.org/en/3.0/nightly/installation/index.html
>
> ** I still see Ansible as the right cross-distro installer as we approach
> the GA date. @ichimonji10 I  am still +1 on your proposal, I think we just
> need to consolidate both dev and testing effort for now. This is similar to
> the approach for the migration tool which we know is really important but
> we aren't starting yet.
>
> -Brian
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev