Re: New feature for 0.6.103

[Andreas Tille]

On Tue, Apr 10, 2018 at 09:18:15PM +0200, Ole Streicher wrote:
> Andreas Tille  writes:
> > I'm aware of this difference and I tried both releases testing and
> > unstable.  Both show differences to the apt cache approach I have no
> > explanation for.  Does my test work for you in Debian Astro?
> 
> The UDD has now the problem with files being in non-free + contrib,
> will adjust that.

I have spotted at least two packages from non-free in the said diff  so
that seems to be at least part of the problem.  (Did I said in my other
mail that the UDD queries tend to get complex? ;-) )

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: New feature for 0.6.103

[Andreas Tille]

On Tue, Apr 10, 2018 at 05:41:01PM +0200, Ole Streicher wrote:
> > I now tried
> >
> > /usr/share/blends-dev/blend-gen-control --udd -r unstable -S  -c -m
> >
> > which results in a change of the debian/control file.  I see some diffs
> > compared to the non-udd version.  I do not have time to investigate those
> > single differences (all in all 40 diff paragraphs).  Just to make sure
> > I understood the test case correctly:  Both results (with --udd and
> > without) should have no difference, right?
> 
> In principle, yes. However the main difference is that the apt version
> takes the packages according the distribution setting in
> /etc/blends/sources.list.*, while the UDD version takes it from the
> release database.
> 
> So, /etc/blends/sources.list.unstable maps to testing, while "--udd -r
> unstable" will query for the unstable release and so take sid.

I'm aware of this difference and I tried both releases testing and
unstable.  Both show differences to the apt cache approach I have no
explanation for.  Does my test work for you in Debian Astro?
 
> I have seen the discussion why unstable maps to testing; and in
> principle one could map "UNRELEASED" --> "unstable" and "unstable" -->
> "testing"; however I don't want to have this coded in two different
> places: once in /etc/blends/sources.list.* and once in
> blends-gen-control. So, if we have it in blends-gen-control (or in the
> Makefile), /etc/sources.list.* should be reverted to standard.

Fine for me.
 
> > BTW,  I think my previous test - as non-sense it probably was to use
> > UNRELEASED - uncovered a problem:  If there is an empty list returned
> > the debian/control file should be changed to everything Suggests.
> > However it did not change at all which is wrong.
> 
> I'll check this.

Thanks

 Andreas. 

-- 
http://fam-tille.de

Re: New feature for 0.6.103

[Ole Streicher]

Andreas Tille  writes:
> I'm not sure about the UDD usage.  I've manually used 
>
>   /usr/share/blends-dev/blend-gen-control --udd -r UNRELEASED -S -t
>
> but it seems its not doing anything. 

There are no double-checks in the moment. The release is directly
queried from the "release" database, so "UNRELEASED" will return an
empty list (and therefore downgrade all packages). I however don't
understand what this is for? Is it a hardcoded equivalent to unstable?

> I also assumed that I do not need
> to do anything manually but could do something like

> +GENCONTROL_OPTS = --udd

> But this does not have any effect.  Am I missing something?

It still does not check the environment. Will do...

Cheers

Ole

Re: New feature for 0.6.103

[Andreas Tille]

Hi Ole,

On Tue, Apr 10, 2018 at 03:17:08PM +0200, Ole Streicher wrote:
> Andreas Tille  writes:
> > I think we should stick to the well tested UDD queries.  It was quite
> > some work to write these.  The only change I have in mind is to restrict
> > the version query to unstable, testing, stable, oldstable and
> > exprimental (currently it returns everything in UDD).  May be we should
> > also return only one version per release (sometimes kfreebsd or hurd
> > is lagging behind and we have several entries for sid).
> 
> I must confess that I find them by far too complicated and hard to
> maintain, but this is another Baustelle yet.

I've learned that the problem is complex.  You can have the complexity
inside a single (or a small set) of SQL queries or you do several
smaller queries and deal with the complexity inside the Python code.  In
the beginning I started with the latter approach.  It turned out that
data traffic between the PostgreSQL client and the UDD host should be
reduced to a minimum.  I had *severe* performance issues when doing
several simple queries.  That's the reason why the goal was to get all
data with a single query.  I did not do this just for fun. ;-)
 
> >> So, I will now add a "python3-blends" package, which however needs then
> >> to go though NEW... If then everyone is happy, we can move back to
> >> unstable.
> >
> > Fine for me.
> 
> I pushed the changes for the split. You can now also to a "from blends
> import Deb822List" as long as it is not merged into the python-debian
> package. 

I've fixed tasks_diff accordingly.
 
> The python3-blends package also contains the HTML documentation.
> 
> Could you have a look before I upload? I would otherwise upload
> tomorrow.

I've set "Standards-Version: 4.1.4"

I'm not sure about the UDD usage.  I've manually used 

  /usr/share/blends-dev/blend-gen-control --udd -r UNRELEASED -S -t

but it seems its not doing anything.  I also assumed that I do not need
to do anything manually but could do something like

med(master) $ git diff Makefile 
diff --git a/Makefile b/Makefile
index e05adcc..e5ae206 100755
--- a/Makefile
+++ b/Makefile
@@ -2,6 +2,8 @@
 
 BLENDMAKEFILE=/usr/share/blends-dev/Makefile
 
+GENCONTROL_OPTS = --udd
+
 CheckBLENDMakefile := $(shell if [ -e $(BLENDMAKEFILE) ] ; then echo 1 ; else 
echo 0 ; fi)
 ifeq ($(CheckBLENDMakefile),1)
 include $(BLENDMAKEFILE)


But this does not have any effect.  Am I missing something?

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: New feature for 0.6.103

[Ole Streicher]

Hi Andreas and all,

Andreas Tille  writes:
> I think we should stick to the well tested UDD queries.  It was quite
> some work to write these.  The only change I have in mind is to restrict
> the version query to unstable, testing, stable, oldstable and
> exprimental (currently it returns everything in UDD).  May be we should
> also return only one version per release (sometimes kfreebsd or hurd
> is lagging behind and we have several entries for sid).

I must confess that I find them by far too complicated and hard to
maintain, but this is another Baustelle yet.

>> So, I will now add a "python3-blends" package, which however needs then
>> to go though NEW... If then everyone is happy, we can move back to
>> unstable.
>
> Fine for me.

I pushed the changes for the split. You can now also to a "from blends
import Deb822List" as long as it is not merged into the python-debian
package. 

The python3-blends package also contains the HTML documentation.

Could you have a look before I upload? I would otherwise upload
tomorrow.

Cheers

Ole

Re: New feature for 0.6.103

[Andreas Tille]

On Tue, Apr 10, 2018 at 01:14:06PM +0200, Ole Streicher wrote:
> > IMO blends-dev should depend on python3-psycopg2 because otherwise the 
> > '-U' flag doesn't seem to work...
> 
> IMO "Recommends" or even "Suggests" would be better, since you need it
> only when using the "-U" flag. I added a note to the option help

I'd vote for Recommends which in a default installations enables all
available options and only --no-recommends leads to a minimized system.

Kind regards

   Andreas. 

-- 
http://fam-tille.de

Re: New feature for 0.6.103

[Ole Streicher]

Andreas Tille  writes:
> On Tue, Apr 10, 2018 at 11:30:07AM +0200, Ole Streicher wrote:
>> > $ git clone https://ole.ath.cx/blends/
>> > Klone nach 'blends' ...
>> > fatal: repository 'https://ole.ath.cx/blends/' not found
>> 
>> My wording was easy to misunderstand. I mean:
>> 
>> $ firefox https://ole.ath.cx/blends/
>> 
>> and then check what you read ;-)
>
> Ahhh- I admit I did this as well.  Please note that this page throws an
> error behind some proxy-firewall-whatever combination.  I needed to use
> my LTS-bound laptop to view the page.

They are just a static copy, so I don't know what could be wrong here.

> When the header does not contain a line
> Format: https://blends.debian.org/blends/1.1
> then the priorities will be lowered when read:
>   □ Recommends –> Suggests
>   □ Depends –> Recommends
>
> May be this is not yet adapted to what I wrote before but this is not
> correct.

I forgot to change the docs here. Fixed in the repo.

>> > I admit this json is of different nature and could be used in connection
>> > with any different method to obtain the Git-based data.  I'm just
>> > afraid about changing something that is *really* rarely used and simply
>> > works just for the reason to make it more elegant.
>> 
>> Sure. I am however not sure whether this is true for generating the
>> changelog entry. Maybe I just add this to blends-gen-control.
>
> May be its a compromise to do the changelog generation based on the
> Git tags.  Probably it would not be a big issue to either hide the
> json files in a separate branch - or may be only exclude these from
> the generation of the source tarball.

Why should one do this? Additional files for local purposes can always
be in the sources.

>> BTW, are you happy with the UDD support? Or do you think we need more
>> here?
>
> I admit I do not fully understand what you mean by "initial package
> list".

I don't want to query all packages, but only those which are in the
blend. Therefore, one needs to give this list, which is then used as
input.

I however meant the blends-gen-control utility itself, not the package.

>  Please note that for the web pages I've crafted quite complex
> queries which gather way more data than will fit into an ordinary
> apt.Cache.  So if the paragraph
>
>blends.uddcache(release, packages, **db_args)¶
>
> is just for the metapackage creation and if it ends up in arch dependant
> metapackages I'm perfectly happy.  But I can not really tell whether we
> need more here from the information I have.

Yes, it is for the metapackage creation only in the moment, and it
contains the versions for all archs (but only for the selected release).

For the web page, this needs probably to be extended to return the
packages for all (or a list of) Debian releases.

Also, for the web page this is however only what we (probably) need to
gather from the "packages" table. One would (in a similar manner) need
to select from popcon etc. And probably you want to create the Blends
infrastructure itself ("Blend", "Task" objects) from UDD instead of from
the files.

So, there is clearly some work for the web pages; the main idea in
blends-dev is to create some common infrastructure so that blends
related stuff does not need to be developed from scratch again.

So, I will now add a "python3-blends" package, which however needs then
to go though NEW... If then everyone is happy, we can move back to
unstable.

Cheers

Ole

Re: New feature for 0.6.103

[Wolfgang Schweer]

On Tue, Apr 10, 2018 at 11:30:07AM +0200, Ole Streicher wrote:
> BTW, are you happy with the UDD support?

IMO blends-dev should depend on python3-psycopg2 because otherwise the 
'-U' flag doesn't seem to work...

Wolfgang

P.S.
I can confirm that own sources.list.* take precedence like expected, 
thanks. 


signature.asc
Description: PGP signature

Re: New feature for 0.6.103

[Andreas Tille]

On Tue, Apr 10, 2018 at 11:30:07AM +0200, Ole Streicher wrote:
> > $ git clone https://ole.ath.cx/blends/
> > Klone nach 'blends' ...
> > fatal: repository 'https://ole.ath.cx/blends/' not found
> 
> My wording was easy to misunderstand. I mean:
> 
> $ firefox https://ole.ath.cx/blends/
> 
> and then check what you read ;-)

Ahhh- I admit I did this as well.  Please note that this page throws an
error behind some proxy-firewall-whatever combination.  I needed to use
my LTS-bound laptop to view the page.



When the header does not contain a line

Format: https://blends.debian.org/blends/1.1

then the priorities will be lowered when read:

  □ Recommends –> Suggests
  □ Depends –> Recommends


May be this is not yet adapted to what I wrote before but this is not
correct.  In Format 1.0  Recommends and Depends are synonyms.  If a
dependency exists both end up in Recommends if not both end up in
Suggests.  Its just that a 1.0 format task file never can produce any
"Depends" in the control file output.
 
> >> We already have git, so creating a function that returns a
> >> `blends.Blend` object based on some git tag (rep. Debian version) is
> >> straightforward with `git worktree` and also (since the blends source
> >> packages are rather small) quick. And has the advantage that you don't
> >> need to maintain an additional json file. You can just generate it on
> >> the fly ;-)
> >
> > I confirm that I understood that it is possible to do this.  However, I
> > can not imagine that whatever kind of diff between tags will be used
> > this is faster than just reading a json file.
> 
> It is not faster; it is probably just not much slower.

Its definitely slower when you put the development and testing time into
consideration. :-P
 
> > In addition I would need to "fake commit" pre-version control data
> > into Git (yes, there were releases of Debian Med before I started
> > using SVN).  These are now simply stored in json that's aggregating
> > the data I want to display[1].
> 
> That is ofcourse a good argument, and also maybe the blends class cannot
> read all the really old data.

I would not mind to fake those commits if the original packages would
be available any more.  But some of these even pre-date snapshot.d.o.
 
> > I admit this json is of different nature and could be used in connection
> > with any different method to obtain the Git-based data.  I'm just
> > afraid about changing something that is *really* rarely used and simply
> > works just for the reason to make it more elegant.
> 
> Sure. I am however not sure whether this is true for generating the
> changelog entry. Maybe I just add this to blends-gen-control.

May be its a compromise to do the changelog generation based on the
Git tags.  Probably it would not be a big issue to either hide the
json files in a separate branch - or may be only exclude these from
the generation of the source tarball.
 
> BTW, are you happy with the UDD support? Or do you think we need more
> here?

I admit I do not fully understand what you mean by "initial package
list".  Please note that for the web pages I've crafted quite complex
queries which gather way more data than will fit into an ordinary
apt.Cache.  So if the paragraph

   blends.uddcache(release, packages, **db_args)¶

is just for the metapackage creation and if it ends up in arch dependant
metapackages I'm perfectly happy.  But I can not really tell whether we
need more here from the information I have.

Thanks a lot again - I'm really happy that your are doing that stuff

  Andreas.


-- 
http://fam-tille.de

Re: New feature for 0.6.103

[Ole Streicher]

Andreas Tille  writes:
> On Tue, Apr 10, 2018 at 10:10:39AM +0200, Ole Streicher wrote:
>> >> If you like, I could try that as next step; then I would however just
>> >> create a "python3-blends" package (to not interfer with the
>> >> python3-debian package).
>> >
>> > May be a python3-blends package assembling (and documenting!) that kind
>> > of tools (and put them on a solid technical base rather than hackish
>> > scripts) would make sense.
>> 
>> That is the main idea of the rewrite.
>> 
>> Check out https://ole.ath.cx/blends/ which is generated from the
>> blends-gen-dev source. Does it fit your needs in terms of functionality
>> and documentation?
>
> $ git clone https://ole.ath.cx/blends/
> Klone nach 'blends' ...
> fatal: repository 'https://ole.ath.cx/blends/' not found

My wording was easy to misunderstand. I mean:

$ firefox https://ole.ath.cx/blends/

and then check what you read ;-)

>> > But I think, sticking to the approach to store the dependency data in
>> > some sensible way (not necessarily inside the source package) is
>> > reasonable if you want to look quickly at historic data.
>> 
>> We already have git, so creating a function that returns a
>> `blends.Blend` object based on some git tag (rep. Debian version) is
>> straightforward with `git worktree` and also (since the blends source
>> packages are rather small) quick. And has the advantage that you don't
>> need to maintain an additional json file. You can just generate it on
>> the fly ;-)
>
> I confirm that I understood that it is possible to do this.  However, I
> can not imagine that whatever kind of diff between tags will be used
> this is faster than just reading a json file.

It is not faster; it is probably just not much slower.

> In addition I would need to "fake commit" pre-version control data
> into Git (yes, there were releases of Debian Med before I started
> using SVN).  These are now simply stored in json that's aggregating
> the data I want to display[1].

That is ofcourse a good argument, and also maybe the blends class cannot
read all the really old data.

> I admit this json is of different nature and could be used in connection
> with any different method to obtain the Git-based data.  I'm just
> afraid about changing something that is *really* rarely used and simply
> works just for the reason to make it more elegant.

Sure. I am however not sure whether this is true for generating the
changelog entry. Maybe I just add this to blends-gen-control.

BTW, are you happy with the UDD support? Or do you think we need more
here?

Cheers

Ole

Re: New feature for 0.6.103

[Ole Streicher]

Andreas Tille  writes:
>> However, tasks_diff may also benefit from a blends
>> Python package, and it could make the handling much easier (no json), as
>> if the blends package is in git with a standard tagging:
>> 
>> * git worktree could check out the latest debian/ tag to some
>>   temporary dir
>> 
>> * one can create a blends.Blend object from that dir
>> 
>> * one can create a blends.Blend object from the actual dir
>> 
>> * compare them, write out, and you are done.
>
> That's surely an option for the changelog creation.  But the json files
> also enable creating statistics[1] using this script[2].  Its just a
> hack and not documented (if I only had the time to do this) but just
> parsing a set of json files is probably way more sensible than to do
> your algorithm over and over again for all available tags.

Another reason to create a separare python3-blends package... that
script should be convertible to use it.

>> If you like, I could try that as next step; then I would however just
>> create a "python3-blends" package (to not interfer with the
>> python3-debian package).
>
> May be a python3-blends package assembling (and documenting!) that kind
> of tools (and put them on a solid technical base rather than hackish
> scripts) would make sense.

That is the main idea of the rewrite.

Check out https://ole.ath.cx/blends/ which is generated from the
blends-gen-dev source. Does it fit your needs in terms of functionality
and documentation?

> But I think, sticking to the approach to store the dependency data in
> some sensible way (not necessarily inside the source package) is
> reasonable if you want to look quickly at historic data.

We already have git, so creating a function that returns a
`blends.Blend` object based on some git tag (rep. Debian version) is
straightforward with `git worktree` and also (since the blends source
packages are rather small) quick. And has the advantage that you don't
need to maintain an additional json file. You can just generate it on
the fly ;-)

Cheers

Ole

Re: New feature for 0.6.103

[Andreas Tille]

Hi Ole,

On Tue, Apr 10, 2018 at 08:42:42AM +0200, Ole Streicher wrote:
> > I admit I would love to re-use it for tasks_diff - so if you could find
> > some temporary solution how I could somehow cleanly get Deb822List that
> > would help but it should be no real blocker.
> 
> Deb822List is a special thing, since it should IMO not go to a
> (debian.)blends package, but to debian.deb822. What prevents me from
> writing a Merge Request on salsa is that the constructor should either
> take a (unicode resp. normal) string, a file-like object, or a list of
> deb822.Deb822 or dicts to be a universal class.
> 
> For the moment, I would recommend just to cut Deb822List from
> blends-gen-control.

OK, fine for me.

> However, tasks_diff may also benefit from a blends
> Python package, and it could make the handling much easier (no json), as
> if the blends package is in git with a standard tagging:
> 
> * git worktree could check out the latest debian/ tag to some
>   temporary dir
> 
> * one can create a blends.Blend object from that dir
> 
> * one can create a blends.Blend object from the actual dir
> 
> * compare them, write out, and you are done.

That's surely an option for the changelog creation.  But the json files
also enable creating statistics[1] using this script[2].  Its just a
hack and not documented (if I only had the time to do this) but just
parsing a set of json files is probably way more sensible than to do
your algorithm over and over again for all available tags.
 
> If you like, I could try that as next step; then I would however just
> create a "python3-blends" package (to not interfer with the
> python3-debian package).

May be a python3-blends package assembling (and documenting!) that kind
of tools (and put them on a solid technical base rather than hackish
scripts) would make sense.  But I think, sticking to the approach to
store the dependency data in some sensible way (not necessarily inside
the source package) is reasonable if you want to look quickly at
historic data.

Kind regards

 Andreas.

[1] 
https://people.debian.org/~tille/talks/20180210_debian-med-sprint/sprints+mentoring.pdf#page=11
[2] 
https://salsa.debian.org/blends-team/website/blob/master/misc/team_analysis_tools/count-dependencies.py

-- 
http://fam-tille.de