Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Fri, Apr 06, 2018 at 02:22:40PM +0200, Steffen Möller wrote:
> Whenever you find the moment, please review the presentation of
> 
> rate4site

$ yamllint metadata 
metadata
  1:1   warning  missing document start "---"  (document-start)
  7:8   errortrailing spaces  (trailing-spaces)
  10:2  errorsyntax error: could not find expected ':'


> sambamba

$ git diff
diff --git a/debian/upstream/metadata b/debian/upstream/metadata
index ee8300c..b4afe9d 100644
--- a/debian/upstream/metadata
+++ b/debian/upstream/metadata
@@ -14,7 +14,7 @@ Reference:
   10.1093/bioinformatics/btv098"
  eprint: "https://academic.oup.com/bioinformatics/article-pdf/\
   31/12/2032/568750/btv098.pdf"
-Registries:
+Registry:
  - Name: bio.tools
Entry: Sambamba
  - Name: OMICtools

> on the task page which do not show their references.

I would love if someone else than me would take over proof-reading this
kind of issues before the code is suspected to be wrong (= I will not
check the other two packages of your other mail right now).  Thanks for
anybody who helps in enhancing metadata but I would love to concentrate
on bug fixing of packages.

Thank you

  Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 4/6/18 2:22 PM, Steffen Möller wrote:

Hi Andreas,

Whenever you find the moment, please review the presentation of

rate4site
sambamba


genometools
maffilter



on the task page which do not show their references.

Cheers,

Steffen

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

Hi Andreas,

Whenever you find the moment, please review the presentation of

rate4site
sambamba

on the task page which do not show their references.

Cheers,

Steffen

Re: d/u/metadata for binary or source tree Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Wed, Apr 04, 2018 at 01:56:42PM +0200, Steffen Möller wrote:
> 
> > https://blends.debian.org/blends/apa.html#datagathering
> > 
> > (A.7.) could be a first shot on this problem.
> One half done. The invitation to contribute to salsa

I have no idea what you might consider *Blend* *specific* in using
Salsa.  Everything worked before Salsa and the fact that Git
repositories now can be changed via web form has nothing to do with
Blends.  Am I missing something?

> and the screenshots and translations is missing.

That's perfectly generic Debian stuff.  Well, I admit I wrote the
UDD importers for screenshots and translations because I intended
to use it on the tasks pages.  But it is used in very different
places for all Debian packages.  So what exactly do you want to be
documented here?  (Patches welcome)

> > That's not about branches.  As previously said:  Registry data are per
> > source package currently.  There is no means in the registry data table
> > to map an entry to a binary package.  Thus autodock *and* autogrid are
> > missing registry data both.
> 
> There was one d/u/metadata file that assigned different references IIRC to
> different binaries of that source package by inventing a sub-hierarchy.

Can you please re-read

   https://lists.debian.org/debian-med/2018/03/msg00119.html

Seek for "For citations we are using the field Debian-package"

> This
> seems to indicate that this is not only an issue for the registry entries.
> How about the following:
> 
>  * d/u/metadata always refers to the source tree as a whole

That's the case.

>  * d/u/metadata also refers to the binary with the same name as the source
> tree.

That's also the case (more or less unintended but it is working like
this).
 
>  * information in d/u/metadata associated with a particular set of binaries
> with the same or different names shall be listed in a "binary: " attribute

See my posting.
 
> Admittedly, I see some issues with that. The notion of a reference binary
> for a package with many binaries would be helpful for Debian in general.
> This would prevent something like the Massxpert entry on our task list that
> describes itself as a transition package.

Hmmm, I do not understand what you mean.  Fixing the massxpert package
is only one commit away:

   
https://salsa.debian.org/blends-team/med/commit/6451da2d816f762cf391f048676c79eee1c25431

Please, if anybody notices this, just fix the according task instead of
assuming intentional displaying cruft.  Its not *that* hard to do to
replace an outdated package name by a correct one.

> But this also means that such
> notion about, say, "end-user relevant" packages vs technical helper packages
> because of arch-independency or libraries etc, should be declared in
> d/control.

I guess you want to invent some extra layer of complexity for d/control
which will never be accepted to solve a problem for Registry entries
which was just solved for citations.  We just need a *decision* I was
asking for in the end of the mail I was linking to above.
 
> Another concern of mine is that we typically do not tag any data for being
> specific for something. We put it in different files. As such, we would need
> d/u/.metadata.

Having data relevant for *binary* packages in a dir named *upstream*
does not sound very logical for me.
 
> For d/u/edam we had the concept of a summary representing the packages as a
> whole and then subsections for every binary.

I admit I do not fully understand the edam data.  I guess the only
reason that we are not facing issues with these data is that they are
just stored in UDD and the only (non-)use case is some query that
exports the data again.  I doubt that anybody has done some QA on the
output.

> I kind of liked it the way it
> is, but maybe separating that out to different files would also be
> appropriate. Could there be an option to do it either way?

Again:  I think that what we implemented for citations (Debian-Package
field) which is documented in Wiki[1] is absolutely sufficient to solve
the problem we have.  I see no reason to invent something else.  The
only thing that we should think about is the data storage model:  Do we
use another column as in the bibref table or should the importer
translate the data right to binary packages according to the algorithm
above:

   If source package = binary package if nothing else is specified.
   Use Debian-Package as binary package name otherwise.

It depends a bit from the applications that will consume that data.  For
the moment it is only the Blends web sentinel which is rather binary
package centric.  The bibref table layout is rather a hack I created
afterwards after we were running in the perfectly same problem as we
have now with references.  May be a binary package based table layout
is more sensible.

> I cannot judge how much of a hassle it is to fiddle anything like that into
> the UDD. What do you think?

I keep on thinking that the established method is not much of a hassle

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

Hi Tony,

On 4/3/18 1:59 PM, Tony Travis wrote:

On 03/04/18 11:27, Steffen Möller wrote:

[...]
This is a bit of a side-track from the core of this thread. Maybe we
would have Tony describing our achievements one they are manifesting in
Bio-Linux. And if we have a look at
https://salsa.debian.org/dashboard/milestones ? Maybe we can use those
as an anchor for achievements from which news could be generated in an
easier way, not necessarily by us.

[...]

Yes, I would be happy to do that.


Yippee.

I see five bits, there may be more:

A) New bits for Bio-Linux that its new release features

B) New bits for Bio-Linux that shall be coming at some future point

C) New bits for Debian and its derivative distributions that have been 
established


D) New bits for Debian and its derivative distributions that are 
currently being worked on


E) New bits for Debian and its derivatives that are currently being 
discussed on the Sprint(s) or mailing list(s).


In an ideal world we would see a lot moving from E to A and from B to E, 
so the effort to describe this all would be mostly done only once :) I 
know, it will not work, let alone because the audiences are different.


In my PoV there is not too much of a conceptional difference between 
porting to Bio-Linux and having backports within Debian. We need to 
address both. Just who should do all that.


Since Debian Med takes about every opportunity that it is just a regular 
part of Debian, we somehow do not have any home page since there is the 
Debian home page already. As such there is also no such thing like a 
page where to distribute any Debian Med-specific news. Closest is possibly


https://blends.debian.org/med/ and http://debian-med.alioth.debian.org/ .

Nobody truly wants to read through our Wiki page on 
https://wiki.debian.org/DebianMed .


We have http://debianmed.blogspot.de/ with latest news from 2014.

So, I have no immediate idea about how to improve on it all.



I'm now only doing minimal maintenance work on Bio-Linux 8, based on
Ubuntu 14.04 LTS (Trusty) while I start work on Bio-Linux 9, based on
Ubuntu-MATE 18.04 LTS (Bionic). I've got the 18.04 Ubuntu-MATE beta .iso
and I've started work remastering it for Bio-Linux using "customizer":


https://github.com/kamilion/customizer

I cannot judge. The readme reads fine.

I plan to base Bio-Linux 9 on Debian-Med + Bioconda


That is good. Please consider to also install the singularity and docker
clients. And the cwltool.


and would like to
start adding Ubuntu 18.04 (Bionic) versions of Debian-Med packages to
the Debian-Med PPA if nobody objects. I also want to drop ALL the 'bad'
NEBC packages (i.e. the binary-only packages that Bela and Tim used
initially to migrate from Debian testing (Sarge) to Ubuntu 6.06 LTS.

Yip.

I've listened to Andreas: I intend to create a "bio-linux-desktop"
meta-package modelled on the Debian-Med/Bio* tasks that are now being
updated to include some 'missing' packages that were only in Bio-Linux.


We should think about adding the version in Ubuntu and Bio-Linux to show 
up among the versions in the task pages. I would not know how to 
implement that, admittedly.


Best,

Steffen

d/u/metadata for binary or source tree Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 4/3/18 5:09 PM, Andreas Tille wrote:

On Tue, Apr 03, 2018 at 12:27:08PM +0200, Steffen Möller wrote:

There is a daily cron job parsing Salsa directories.

Fine. Somewhere there is (or should be :o) ) a documentation how this page
is crafted. On our Wiki? Let us then have a link to that page.

May be this

https://blends.debian.org/blends/apa.html#datagathering

(A.7.) could be a first shot on this problem.
One half done. The invitation to contribute to salsa and the screenshots 
and translations is missing.



Could branches for your cron job's autodock checkout differ? The page was
updated
this morning but yet not references. Or is there a second directory
"autodock" when
the source package name is "autodocksuite" (because of the joint autogrid
tool)?

That's not about branches.  As previously said:  Registry data are per
source package currently.  There is no means in the registry data table
to map an entry to a binary package.  Thus autodock *and* autogrid are
missing registry data both.


There was one d/u/metadata file that assigned different references IIRC 
to different binaries of that source package by inventing a 
sub-hierarchy. This seems to indicate that this is not only an issue for 
the registry entries. How about the following:


 * d/u/metadata always refers to the source tree as a whole

 * d/u/metadata also refers to the binary with the same name as the 
source tree.


 * information in d/u/metadata associated with a particular set of 
binaries with the same or different names shall be listed in a "binary: 
" attribute


Admittedly, I see some issues with that. The notion of a reference 
binary for a package with many binaries would be helpful for Debian in 
general. This would prevent something like the Massxpert entry on our 
task list that describes itself as a transition package. But this also 
means that such notion about, say, "end-user relevant" packages vs 
technical helper packages because of arch-independency or libraries etc, 
should be declared in d/control.


Another concern of mine is that we typically do not tag any data for 
being specific for something. We put it in different files. As such, we 
would need d/u/.metadata.


For d/u/edam we had the concept of a summary representing the packages 
as a whole and then subsections for every binary. I kind of liked it the 
way it is, but maybe separating that out to different files would also 
be appropriate. Could there be an option to do it either way?


I cannot judge how much of a hassle it is to fiddle anything like that 
into the UDD. What do you think?


Best,

Steffen

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Tue, Apr 03, 2018 at 12:27:08PM +0200, Steffen Möller wrote:
> > There is a daily cron job parsing Salsa directories.
> Fine. Somewhere there is (or should be :o) ) a documentation how this page
> is crafted. On our Wiki? Let us then have a link to that page.

May be this

   https://blends.debian.org/blends/apa.html#datagathering

(A.7.) could be a first shot on this problem.

> Could branches for your cron job's autodock checkout differ? The page was
> updated
> this morning but yet not references. Or is there a second directory
> "autodock" when
> the source package name is "autodocksuite" (because of the joint autogrid
> tool)?

That's not about branches.  As previously said:  Registry data are per
source package currently.  There is no means in the registry data table
to map an entry to a binary package.  Thus autodock *and* autogrid are
missing registry data both.
 
Kind regards

  Andreas. 

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Tony Travis]

On 03/04/18 11:27, Steffen Möller wrote:
> [...]
> This is a bit of a side-track from the core of this thread. Maybe we
> would have Tony describing our achievements one they are manifesting in
> Bio-Linux. And if we have a look at
> https://salsa.debian.org/dashboard/milestones ? Maybe we can use those
> as an anchor for achievements from which news could be generated in an
> easier way, not necessarily by us.

Hi, Steffen.

Yes, I would be happy to do that.

I'm now only doing minimal maintenance work on Bio-Linux 8, based on
Ubuntu 14.04 LTS (Trusty) while I start work on Bio-Linux 9, based on
Ubuntu-MATE 18.04 LTS (Bionic). I've got the 18.04 Ubuntu-MATE beta .iso
and I've started work remastering it for Bio-Linux using "customizer":

> https://github.com/kamilion/customizer

I plan to base Bio-Linux 9 on Debian-Med + Bioconda and would like to
start adding Ubuntu 18.04 (Bionic) versions of Debian-Med packages to
the Debian-Med PPA if nobody objects. I also want to drop ALL the 'bad'
NEBC packages (i.e. the binary-only packages that Bela and Tim used
initially to migrate from Debian testing (Sarge) to Ubuntu 6.06 LTS.

I've listened to Andreas: I intend to create a "bio-linux-desktop"
meta-package modelled on the Debian-Med/Bio* tasks that are now being
updated to include some 'missing' packages that were only in Bio-Linux.

Thanks again to all in the Debian-Med team for your help and support,

  Tony.

-- 
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
Registered Office: 3 Donview, Bridge of Alford, AB33 8QJ, Scotland (UK)
tel. +44(0)19755 63548http://minke-informatics.co.uk
mob. +44(0)7985 078324mailto:tony.tra...@minke-informatics.co.uk

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 4/3/18 10:30 AM, Andreas Tille wrote:

On Tue, Apr 03, 2018 at 12:14:57AM +0200, Steffen Möller wrote:

Registry:
   - Name: OMICtools
 Entry: NA
   - Name: RRID
 Entry: NA
   - Name: bio.tools
 Entry: NA

That is an interesting one. Please kindly check

https://salsa.debian.org/med-team/autodocksuite/blob/master/debian/upstream/metadata

which on my side shows

Registry:
  - Name: OMICtools
    Entry: OMICS_19997
  - Name: SciCrunch
    Entry: SCR_012746
  - Name: bio.tools
    Entry: AutoDock

May be you forgot to push.  I now received

commit e5292af2136df30df8ea2a0da0dbeba6b82b027b (HEAD -> master, origin/master)
Author: Steffen Möller 
Date:   Mon Mar 26 16:05:33 2018 +

 Added RRID to metadata

which has the values you are mentioning.  This was not available in the public
repository (at least until 1st of April).

Hm. No. Have not touched it again. May be some delay-thingy somewhere.

I remember it was not your prefered solution but for the moment 'NA'
values are not stored. [...]

How would you like Name and no fancy colouring?

The idea was to inform the world (and ourselves) that we have checked.

I admit I do not like this.  The thing is that the maintainers of the
repository could have updated their data.  Recent development shows that
this is a very probable action these days.  I do not think that the fact
that we have checked is no information which is valuable for a random
visitor of the tasks page and it might be simply wrong since the data
were updated.  So I keep on thinking that 'NA' is not anything
interesting for the page.  For us as developers we can easily do

 grep -w 'NA' */debian/upstream/metadata

if we want to find the information.


Right. If we developers have the whole of Debian Med checked out. I don't.
It is a side-issue. Don't worry.


The
problem is that we are unlikely to be informed about an entry being added to
the registry, so we would look bad. But since the advent of salsa.d.o I am
tempted to risk that.

Sorry, I do not understand this.


We have that mistake only for a few days upon notification.


  

bio-express (https://salsa.debian.org/med-team/express)

It is rather berkeley-express.  This works now[1].  The reason for the
problem was that the repository name is different from the source
package name.  The importer was simply wrong for these cases.  This is
fixed now (hopefully!) but we should definitely avoid this kind of
divergence and I'm tempted to adapt the repository name to the source
package name once we migrate anonscm.d.o to salsa.d.o (I've lost hope
for some sensible solution to keep anonscm working :-((().

Aaah. Yes. This sounds like a reasonable addition to the Debian Med policy.

We are intuitively doing it in close to all of our packages but
specifying it explicitly in policy makes sense, definitely.


gromacs

No Registry entries in d/upstream/metadata - so nothing to display here.

:o/ Because that is debichem and looked at my not-yet-merged own branch.

Why not asking for membership in Debichem and commit directly?


I was with debichem on alioth and eventually will ask for membership again.
However, I would loose the external view on how we present ourselves. And I
am much after attracting casual contributors to our cause, e.g. to improve
the description of our packages or ... registry entries in d/u/metadata. So,
debichem is a test-case to teach me how this feels on the other side.





This document is prepared every 24 hours for Debian packages selected in the
https://salsa.debian.org/blends-team/med/blob/master/tasks/bio;>Med
Bio Task Description withinformation in the Ultimate Debian Database (https://udd.debian.org;>UDD).

The bottom line has a creation date of the web page.  Currently it says:

  Last update: Mon, 02 Apr 2018 19:21:37 -

Excellent. I have missed that one.

It was last updated at [Date+Time].

This information is not available, sorry.

I am uncertain about what "this" refers to, but maybe you have some idea
about who the page can explain itself to the ones who want to contribute
code or content.

There is a daily cron job parsing Salsa directories.

Fine. Somewhere there is (or should be :o) ) a documentation how this page
is crafted. On our Wiki? Let us then have a link to that page. Casual 
contributors,
that is what I am primarily after. For everything else I happily ask you 
or Ole.

I do not keep
track on the information when this job is finished.  The UDD importer is
reading the result of this job later and I also do not keep track when
this job ends.  So we have two uncertain times - the only time that can
be safely displayed is when the web pages are displayed which is done on
the bottom line.  I admit that I'm not very motivated to make some
effort to keep track of the other times compared to other tasks on my
desk.


That is fine.

There could be a line like "Please allow 48 hours for your change to the
package selection or the package description to 

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Tue, Apr 03, 2018 at 12:14:57AM +0200, Steffen Möller wrote:
> 
> > Registry:
> >   - Name: OMICtools
> > Entry: NA
> >   - Name: RRID
> > Entry: NA
> >   - Name: bio.tools
> > Entry: NA
> 
> That is an interesting one. Please kindly check
> 
> https://salsa.debian.org/med-team/autodocksuite/blob/master/debian/upstream/metadata
> 
> which on my side shows
> 
> Registry:
>  - Name: OMICtools
>    Entry: OMICS_19997
>  - Name: SciCrunch
>    Entry: SCR_012746
>  - Name: bio.tools
>    Entry: AutoDock

May be you forgot to push.  I now received

commit e5292af2136df30df8ea2a0da0dbeba6b82b027b (HEAD -> master, origin/master)
Author: Steffen Möller 
Date:   Mon Mar 26 16:05:33 2018 +

Added RRID to metadata

which has the values you are mentioning.  This was not available in the public
repository (at least until 1st of April).
 
> > I remember it was not your prefered solution but for the moment 'NA'
> > values are not stored. [...]
> How would you like Name and no fancy colouring?
> 
> The idea was to inform the world (and ourselves) that we have checked.

I admit I do not like this.  The thing is that the maintainers of the
repository could have updated their data.  Recent development shows that
this is a very probable action these days.  I do not think that the fact
that we have checked is no information which is valuable for a random
visitor of the tasks page and it might be simply wrong since the data
were updated.  So I keep on thinking that 'NA' is not anything
interesting for the page.  For us as developers we can easily do

grep -w 'NA' */debian/upstream/metadata

if we want to find the information.

> The
> problem is that we are unlikely to be informed about an entry being added to
> the registry, so we would look bad. But since the advent of salsa.d.o I am
> tempted to risk that.

Sorry, I do not understand this.
 
> > > bio-express (https://salsa.debian.org/med-team/express)
> > It is rather berkeley-express.  This works now[1].  The reason for the
> > problem was that the repository name is different from the source
> > package name.  The importer was simply wrong for these cases.  This is
> > fixed now (hopefully!) but we should definitely avoid this kind of
> > divergence and I'm tempted to adapt the repository name to the source
> > package name once we migrate anonscm.d.o to salsa.d.o (I've lost hope
> > for some sensible solution to keep anonscm working :-((().
> Aaah. Yes. This sounds like a reasonable addition to the Debian Med policy.

We are intuitively doing it in close to all of our packages but
specifying it explicitly in policy makes sense, definitely.

> > > gromacs
> > No Registry entries in d/upstream/metadata - so nothing to display here.
> :o/ Because that is debichem and looked at my not-yet-merged own branch.

Why not asking for membership in Debichem and commit directly?

> > > This document is prepared every 24 hours for Debian packages selected in 
> > > the
> > >  > > href="https://salsa.debian.org/blends-team/med/blob/master/tasks/bio;>Med
> > > Bio Task Description withinformation in the Ultimate Debian Database 
> > > ( > > href="https://udd.debian.org;>UDD).
> > The bottom line has a creation date of the web page.  Currently it says:
> > 
> >  Last update: Mon, 02 Apr 2018 19:21:37 -
> Excellent. I have missed that one.

> > > It was last updated at [Date+Time].
> > This information is not available, sorry.
> 
> I am uncertain about what "this" refers to, but maybe you have some idea
> about who the page can explain itself to the ones who want to contribute
> code or content.

There is a daily cron job parsing Salsa directories.  I do not keep
track on the information when this job is finished.  The UDD importer is
reading the result of this job later and I also do not keep track when
this job ends.  So we have two uncertain times - the only time that can
be safely displayed is when the web pages are displayed which is done on
the bottom line.  I admit that I'm not very motivated to make some
effort to keep track of the other times compared to other tasks on my
desk.
 
> > There is no difference between pushing from remote repository and a web
> > interface edit.
> 
> Good.
> 
> Curious about this autodock/autodocksuite thingy and maybe there is some
> chance to have this self-explaining bit at the end.

I have no idea how this can happen.  I do not use the web interface for
editing package data.
 
> I have added registry links to about all green entries in bio now. Let us
> review those over the next weeks a bit and then think about an announcing it
> with the Debian News or whatever the list finds appropriate.

I'd love if we could find some default channel for Debian Med news (and
a person who feels dedicated to feed this channel regularly). 

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 4/2/18 11:32 PM, Andreas Tille wrote:

Hi Steffen,

thanks a lot for your continuous work on assembling registry data.

On Sat, Mar 31, 2018 at 02:41:42AM +0200, Steffen Möller wrote:

There are still packages that are not updated. Worthwhile candidates
are

autodock

This has:

Registry:
  - Name: OMICtools
Entry: NA
  - Name: RRID
Entry: NA
  - Name: bio.tools
Entry: NA


That is an interesting one. Please kindly check

https://salsa.debian.org/med-team/autodocksuite/blob/master/debian/upstream/metadata

which on my side shows

Registry:
 - Name: OMICtools
   Entry: OMICS_19997
 - Name: SciCrunch
   Entry: SCR_012746
 - Name: bio.tools
   Entry: AutoDock



I remember it was not your prefered solution but for the moment 'NA'
values are not stored. [...]

How would you like Name and no fancy colouring?

The idea was to inform the world (and ourselves) that we have checked. 
The problem is that we are unlikely to be informed about an entry being 
added to the registry, so we would look bad. But since the advent of 
salsa.d.o I am tempted to risk that.



bio-express (https://salsa.debian.org/med-team/express)

It is rather berkeley-express.  This works now[1].  The reason for the
problem was that the repository name is different from the source
package name.  The importer was simply wrong for these cases.  This is
fixed now (hopefully!) but we should definitely avoid this kind of
divergence and I'm tempted to adapt the repository name to the source
package name once we migrate anonscm.d.o to salsa.d.o (I've lost hope
for some sensible solution to keep anonscm working :-((().

Aaah. Yes. This sounds like a reasonable addition to the Debian Med policy.



Bio-eagle

Works now[2].  Same as above.

Thanks.



garli

Works (and as far as I can see it has also worked before - no idea why
it is on your list).

Most likely I did not wait long enough prior to reporting.

gdpc

Same here - just works.


gromacs

No Registry entries in d/upstream/metadata - so nothing to display here.

:o/ Because that is debichem and looked at my not-yet-merged own branch.

a problem for me to report them is that I am uncertain about the time that
needs
to be passed until the update is run. Could you please help with a little
line at the bottom alike:

This document is prepared every 24 hours for Debian packages selected in the
https://salsa.debian.org/blends-team/med/blob/master/tasks/bio;>Med
Bio Task Description withinformation in the Ultimate Debian Database (https://udd.debian.org;>UDD).

The bottom line has a creation date of the web page.  Currently it says:

 Last update: Mon, 02 Apr 2018 19:21:37 -

Excellent. I have missed that one.

It was last updated at [Date+Time].

This information is not available, sorry.


I am uncertain about what "this" refers to, but maybe you have some idea 
about who the page can explain itself to the ones who want to contribute 
code or content.






The good part of the problem is that whenever I check if something was
updated, I'll add RRIDs to another package. This typically is from within
salsa. Is that possibly not triggering the same routines as when I push from
a remote-from-salsa repository?

There is no difference between pushing from remote repository and a web
interface edit.


Good.

Curious about this autodock/autodocksuite thingy and maybe there is some 
chance to have this self-explaining bit at the end.


I have added registry links to about all green entries in bio now. Let 
us review those over the next weeks a bit and then think about an 
announcing it with the Debian News or whatever the list finds appropriate.


Best,

Steffen

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

Hi Steffen,

thanks a lot for your continuous work on assembling registry data.

On Sat, Mar 31, 2018 at 02:41:42AM +0200, Steffen Möller wrote:
> There are still packages that are not updated. Worthwhile candidates
> are
> 
> autodock

This has:

Registry:
 - Name: OMICtools
   Entry: NA
 - Name: RRID
   Entry: NA
 - Name: bio.tools
   Entry: NA

I remember it was not your prefered solution but for the moment 'NA'
values are not stored.  The importer log says:

2018-04-01 14:57:13,012 - DEBUG - (125): Registry data found for source 
'autodocktools' of debian-med: registry = [{'Entry': 'NA', 'Name': 
'OMICtools'}, {'Entry': 'NA', 'Name': 'RRID'}, {'E
ntry': 'NA', 'Name': 'bio.tools'}]
2018-04-01 14:57:13,013 - INFO - (364): Registry entry OMICtools from source 
autodocktools was removed since it is NA.
2018-04-01 14:57:13,013 - INFO - (364): Registry entry RRID from source 
autodocktools was removed since it is NA.
2018-04-01 14:57:13,013 - INFO - (364): Registry entry bio.tools from source 
autodocktools was removed since it is NA.
2018-04-01 14:57:13,013 - DEBUG - (415): No registry data for source 
'autodocktools' of debian-med: upstream.registry = []


So this works as expected.

> bio-express (https://salsa.debian.org/med-team/express)

It is rather berkeley-express.  This works now[1].  The reason for the
problem was that the repository name is different from the source
package name.  The importer was simply wrong for these cases.  This is
fixed now (hopefully!) but we should definitely avoid this kind of
divergence and I'm tempted to adapt the repository name to the source
package name once we migrate anonscm.d.o to salsa.d.o (I've lost hope
for some sensible solution to keep anonscm working :-((().

> Bio-eagle

Works now[2].  Same as above.

> garli

Works (and as far as I can see it has also worked before - no idea why
it is on your list).

> gdpc

Same here - just works.

> gromacs

No Registry entries in d/upstream/metadata - so nothing to display here.
 
> a problem for me to report them is that I am uncertain about the time that
> needs
> to be passed until the update is run. Could you please help with a little
> line at the bottom alike:
> 
> This document is prepared every 24 hours for Debian packages selected in the
> https://salsa.debian.org/blends-team/med/blob/master/tasks/bio;>Med
> Bio Task Description withinformation in the Ultimate Debian Database ( href="https://udd.debian.org;>UDD).

The bottom line has a creation date of the web page.  Currently it says:

Last update: Mon, 02 Apr 2018 19:21:37 -

> It was last updated at [Date+Time].

This information is not available, sorry.

> The good part of the problem is that whenever I check if something was
> updated, I'll add RRIDs to another package. This typically is from within
> salsa. Is that possibly not triggering the same routines as when I push from
> a remote-from-salsa repository?

There is no difference between pushing from remote repository and a web
interface edit.

Kind regards

   Andreas.

[1] https://blends.debian.org/med/tasks/bio#berkeley-express 
[2] https://blends.debian.org/med/tasks/bio#bio-eagle

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

Hi Andreas,

On 3/30/18 3:14 PM, Andreas Tille wrote:

Hi Steffen,

On Fri, Mar 30, 2018 at 01:11:28AM +0200, Andreas Tille wrote:

Please kindly have a look at clonalframe - it has a ref
for SciCrunch, is yamllint clean but nothing shows up.

H, good catch.

  clonalframe   | SciCrunch | SCR_016060

is in UDD.  I have no idea why it is not displayed on the tasks page.
Needs further investigation.

Fixed now.

Happy here. :)

I realised that importer and tasks pages generation scripts
were all perfectly working.  However, for a reason I did not understand
the gatherer was not run on the main UDD host.  I need to keep an eye on
this.


There are still packages that are not updated. Worthwhile candidates
are

autodock
bio-express (https://salsa.debian.org/med-team/express)
Bio-eagle
garli
gdpc
gromacs

a problem for me to report them is that I am uncertain about the time 
that needs
to be passed until the update is run. Could you please help with a 
little line at the bottom alike:


This document is prepared every 24 hours for Debian packages selected in 
the href="https://salsa.debian.org/blends-team/med/blob/master/tasks/bio;>Med 
Bio Task Description withinformation in the Ultimate Debian Database 
(https://udd.debian.org;>UDD).  It was last updated at 
[Date+Time].


The good part of the problem is that whenever I check if something was 
updated, I'll add RRIDs to another package. This typically is from 
within salsa. Is that possibly not triggering the same routines as when 
I push from a remote-from-salsa repository?


Best,

Steffen

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

Hi Steffen,

On Fri, Mar 30, 2018 at 01:11:28AM +0200, Andreas Tille wrote:
> > Please kindly have a look at clonalframe - it has a ref
> > for SciCrunch, is yamllint clean but nothing shows up.
> 
> H, good catch.  
> 
>  clonalframe   | SciCrunch | SCR_016060
> 
> is in UDD.  I have no idea why it is not displayed on the tasks page.
> Needs further investigation.

Fixed now.  I realised that importer and tasks pages generation scripts
were all perfectly working.  However, for a reason I did not understand
the gatherer was not run on the main UDD host.  I need to keep an eye on
this.

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Thu, Mar 29, 2018 at 08:48:01PM +0200, Steffen Möller wrote:
> 
> > > But I found artemis
> > > (https://salsa.debian.org/med-team/artemis/tree/master/debian/upstream)
> > > which was not touched for a while and looks syntactically just fine but 
> > > has
> > > its RRIDs not shown.
> 
> Please kindly have a look at clonalframe - it has a ref
> for SciCrunch, is yamllint clean but nothing shows up.

H, good catch.  

 clonalframe   | SciCrunch | SCR_016060

is in UDD.  I have no idea why it is not displayed on the tasks page.
Needs further investigation.  And flexbar is also fully read:

 flexbar   | OMICtools | OMICS_01087
 flexbar   | bio.tools | flexbar
 flexbar   | SciCrunch | SCR_013001

I need to check this in the next couple of days. 

Kind regards

   Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 3/29/18 8:48 PM, Steffen Möller wrote:


On 3/28/18 1:14 PM, Andreas Tille wrote:

On Wed, Mar 28, 2018 at 12:03:54PM +0200, Steffen Möller wrote:

But I found artemis
(https://salsa.debian.org/med-team/artemis/tree/master/debian/upstream)
which was not touched for a while and looks syntactically just fine 
but has

its RRIDs not shown.


Please kindly have a look at clonalframe - it has a ref
for SciCrunch, is yamllint clean but nothing shows up.


For flexbar it is only the missing \n that is missing.

Cheers,

Steffen

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

On 3/28/18 1:14 PM, Andreas Tille wrote:

On Wed, Mar 28, 2018 at 12:03:54PM +0200, Steffen Möller wrote:

But I found artemis
(https://salsa.debian.org/med-team/artemis/tree/master/debian/upstream)
which was not touched for a while and looks syntactically just fine but has
its RRIDs not shown.


Please kindly have a look at clonalframe - it has a ref
for SciCrunch, is yamllint clean but nothing shows up.

Many thanks

Steffen

Re: RRID update on salsa on packages starting with A+B

[Dylan Aïssi]

Hi Steffen, hi Andreas,

2018-03-28 14:16 GMT+02:00 Andreas Tille :
>
>> Via salsa, though, well, not. Should lintian invoke yamllint, possibly?
>
> As far as I know syntax errors in upstream files are fetched by lintian.
> Its the downside of just using the not yet uploaded commits as data
> input that we do not run lintian on these.
>

Currently, the lintian checks for integrity of d/u/metadata are
disabled due to some security problem [1] but maybe should be
re-enabled soon [2].

Best,
Dylan

[1] 
https://anonscm.debian.org/git/lintian/lintian.git/commit/checks/upstream-metadata.pm?id=6119d49c3b
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=862373

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Wed, Mar 28, 2018 at 02:10:33PM +0200, Steffen Möller wrote:
> 
> > Checking UDD log
> > 
> > blends_prospective_gatherer.log:2018-03-27 08:32:05,582 - ERROR - (110): 
> > Scanner error in file 
> > /srv/udd.debian.org/mirrors/machine-readable/a/artemis.upstream of 
> > debian-med: mapping values are not allowed here
> > blends_prospective_gatherer.log:   Entry: artemis
> 
> Excellent. Thank you (and yamllint) for spotting that.

You are welcome.
 
> Concerning yamllint I am a routine user whenever I edit via the command
> line. And I also keep fixing the long lines.

Nice. :-)
 
> Via salsa, though, well, not. Should lintian invoke yamllint, possibly?

As far as I know syntax errors in upstream files are fetched by lintian.
Its the downside of just using the not yet uploaded commits as data
input that we do not run lintian on these.

Kind regards

 Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

On Wed, Mar 28, 2018 at 12:03:54PM +0200, Steffen Möller wrote:
> But I found artemis
> (https://salsa.debian.org/med-team/artemis/tree/master/debian/upstream)
> which was not touched for a while and looks syntactically just fine but has
> its RRIDs not shown.

Checking UDD log

blends_prospective_gatherer.log:2018-03-27 08:32:05,582 - ERROR - (110): 
Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/a/artemis.upstream of debian-med: 
mapping values are not allowed here
blends_prospective_gatherer.log:   Entry: artemis


Checking package:

$ yamllint debian/upstream/metadata
debian/upstream/metadata
  1:1   warning  missing document start "---"  (document-start)
  2:81  errorline too long (111 > 80 characters)  (line-length)
  2:111 errortrailing spaces  (trailing-spaces)
  3:81  errorline too long (126 > 80 characters)  (line-length)
  12:81 errorline too long (82 > 80 characters)  (line-length)
  13:81 errorline too long (139 > 80 characters)  (line-length)
  23:81 errorline too long (83 > 80 characters)  (line-length)
  24:81 errorline too long (164 > 80 characters)  (line-length)
  25:81 errorline too long (104 > 80 characters)  (line-length)
  34:81 errorline too long (84 > 80 characters)  (line-length)
  41:9  errorsyntax error: mapping values are not allowed here

Well, yamllint is a bit picky but the mapping values error matches the
UDD importer.

So lets have a look:

$ git diff
diff --git a/debian/upstream/metadata b/debian/upstream/metadata
index 4bede0a..a9906a8 100644
--- a/debian/upstream/metadata
+++ b/debian/upstream/metadata
@@ -37,5 +37,5 @@ Registry:
Entry: SCR_004267
  - Name: OMICtools
Entry: OMICS_00903
- - Name; bio.tools
+ - Name: bio.tools
Entry: artemis

... voila, the bug in question vanished.
 
> Please also have an eye on clonalframe
> (https://salsa.debian.org/med-team/clonalframe/tree/master/debian/upstream)
> which was not updated even though the other changes of mine seem to be all
> in.

Same here.  Importer says:

blends_prospective_gatherer.log:2018-03-27 08:34:50,809 - ERROR - (110): 
Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/c/clonalframe.upstream of 
debian-med: mapping values are not allowed here


$ yamllint debian/upstream/metadata
debian/upstream/metadata
  1:1   warning  missing document start "---"  (document-start)
  21:10 errorsyntax error: mapping values are not allowed here


$ git diff
diff --git a/debian/upstream/metadata b/debian/upstream/metadata
index 79c7b56..98566d5 100644
--- a/debian/upstream/metadata
+++ b/debian/upstream/metadata
@@ -17,5 +17,5 @@ Registry:
 Entry: SCR_016060
   - Name: bio.tools
 Entry: NA
-  - Name. OMICtools
+  - Name: OMICtools
 Entry: NA


So please use yamllint on all your updates - if not on all at least
those that are suspicious since not showing up on the sentinel page.  It
seems just editing in Salsa online is a bit error prone.

>From time to time (every half year or so) I'm doing some QA on the UDD
log but inbetween I will not notice those things.  Here is the full list
of yaml issues in upstream files:

$ grep "ERROR.*\.upstream of" blends_prospective_gatherer.log 
2018-03-27 08:32:05,582 - ERROR - (110): Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/a/artemis.upstream of debian-med: 
mapping values are not allowed here
2018-03-27 08:33:15,614 - ERROR - (110): Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/o/octave-stk.upstream of 
pkg-octave: mapping values are not allowed here
2018-03-27 08:33:17,622 - ERROR - (110): Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/o/octave-divand.upstream of 
pkg-octave: mapping values are not allowed here
2018-03-27 08:34:00,876 - ERROR - (110): Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/r/rapmap.upstream of debian-med: 
mapping values are not allowed here
2018-03-27 08:34:50,809 - ERROR - (110): Scanner error in file 
/srv/udd.debian.org/mirrors/machine-readable/c/clonalframe.upstream of 
debian-med: mapping values are not allowed here


> I'll wait for tomorrow to indicate the others. Anyway, good to hear that no
> extra upload is required.

I can confirm that for this kind of data no upload was required for
years. 

Kind regards

  Andreas.

-- 
http://fam-tille.de

Re: RRID update on salsa on packages starting with A+B

[Steffen Möller]

Hi Andreas,

I'll get to the other points of your fine reply a bit later. The easy 
ones first


On 3/27/18 9:04 AM, Andreas Tille wrote:

Anyway. I came across

  * one or two entries

Which ones?


I had thought this would be dead easy to answer and then I added quite a 
few more references since indeed there was no RRID assigned, yet. Hm. 
Did I sometimes close the salsa editing window prior to pushing the edit?


But I found artemis 
(https://salsa.debian.org/med-team/artemis/tree/master/debian/upstream) 
which was not touched for a while and looks syntactically just fine but 
has its RRIDs not shown.


Please also have an eye on clonalframe 
(https://salsa.debian.org/med-team/clonalframe/tree/master/debian/upstream) 
which was not updated even though the other changes of mine seem to be 
all in.


I'll wait for tomorrow to indicate the others. Anyway, good to hear that 
no extra upload is required.


Steffen

Re: RRID update on salsa on packages starting with A+B

[Andreas Tille]

Hi Steffen,

On Mon, Mar 26, 2018 at 07:23:24PM +0200, Steffen Möller wrote:
> 
> I just procrastinated a bit into using the comfort of salsa to update
> debian/upstream/metadata and here the references to SciCrunch, OMICtools and
> bio.tools registries. All three registries have improved their coverage
> enormously over the past few months. I am deeply impressed.

Thanks a lot for the large update.
 
> Anyway. I came across
> 
>  * one or two entries

Which ones?

> that had perfect RRID descriptions on salsa but not on
> our task page - does the package need to be re-uploaded for the change to
> become visible?

Re-uploading is *not* needed.  The data come from Salsa Git repositories
(since about two weeks the machine-readable gatherer was pointed from
Alioth to Salsa).  However, there is an about 24 hour delay between
commits and visibility of the data on the web sentinel since at least
two cron jobs are involved (one that gathers the data and one that
creates the pages).

>  * belvu and blixem that are from the same source package but have different
> task entries and also separate catalog entries in all three registries. This
> breaks the current UDD schema. I have annotated it now as ['belvu','blixem']
> (for bio.tools, the others analogously).
> 
> Ideas for improvements anyone? Or is this how it should be for now?

I'm not sure.  In any case the current gatherer code will do nothing (at
best) or fail.  It seems that we are lucky and it does not break.  The
thing is that if we change our data model somebody (currently only me)
needs to adapt the code.  Currently there is no chance to resolve

 - Name: OMICtools
   Entry: ['OMICS_23183', 'OMICS_23184', 'OMICS_15828']

or

 - Name: SciCrunch
   Entry: ['SCR_015989','SCR_015994', 'NA']

How should the gatherer magically guess what binary package to choose?
The entry

 - Name: bio.tools
   Entry: ['belvu', 'blixem', 'dotter']

looks helpfull - but it is just pure luck that bio.tools has choosen IDs
matching our package names.  So I think your data model is not helpful
since there is no chance to define a sequence of the binary packages
build from one source package.  Thus we somehow need to define the
binary package name explicitly.

For citations we are using the field Debian-package[1] which is for
instance used for meme package[2] (just to have another example since
in seqtools also the dotter publication is marked like this).  However,
this is because I once added an additional field "package" to the bibref
table which looks for instance like this:


udd=# select * from bibref where (source = 'meme' or source = 'seqtools' ) and 
key = 'title';
  source  |  key  |  value  
 | package | rank 
--+---+--+-+--
 meme | title | MEME: discovering and analyzing DNA and protein sequence 
motifs  | |0
 meme | title | Discovering Sequence Motifs with Arbitrary Insertions and 
Deletions  | glam2   |0
 seqtools | title | SeqTools: visual tools for manual analysis of sequence 
alignments| |0
 seqtools | title | Scoredist: A simple and robust protein sequence distance 
estimator   | |1
 seqtools | title | A dot-matrix program with dynamic threshold control suited 
for genomic DNA and protein sequence analysis | dotter  |0
 seqtools | title | A workbench for large-scale sequence homology analysis  
 | |2


You see, packages with different names than the source packages got an
additional value in the package column since it was defined in our data
model first and implemented in the code afterwards.  However, the
registry table looks like this:


udd=# select * from registry where source = 'seqtools';
  source  |   name|   entry   
--+---+---
 seqtools | OMICtools | {OMICS_23183,OMICS_23184}
 seqtools | bio.tools | {belvu,blixem}
 seqtools | SciCrunch | {SCR_015989,SCR_015994}


That's the status before your last commit since the machine-readable
gatherer cron job was not run yet.  The gatherer takes what it gets and
injects it into the database.  Its not magic - its code that needs to be
adapted to a data model.  Changing the data model and hoping that
something sensible will happen is not working.

What we should clarify in advance is:  Does the source column in the
registry table make sense at all or should it rather be a package column
refering to binary packages?  The web sentinel is working on binary
packages so may be we should not keep source package names but rather
binary package names inside this table.  Alternatively