Andreas Tille pushed to branch experimental at Debian Blends Team / blends
Commits: 3228dff2 by Andreas Tille at 2018-04-03T16:56:27+02:00 Description how Blends relevant data are gathered and stored - - - - - 1 changed file: - doc/en/A_devel.xml Changes: ===================================== doc/en/A_devel.xml ===================================== --- a/doc/en/A_devel.xml +++ b/doc/en/A_devel.xml @@ -911,4 +911,54 @@ the changes by doing: </orderedlist> </para> </sect1> + <sect1 id="datagathering"> + <title>Description how Blends relevant data are gathered and stored</title> +<para> +All data relevant for Blends are available in +<ulink url="https://wiki.debian.org/UltimateDebianDatabase";>Ultimate Debian Database (UDD)</ulink>. +Here is a description of the data gatherers that were explicitly invented to provide +information for usage in Blends websentinel. +</para> + + <sect2 id="ftpnew"> + <title>Packages in Debian ftp new queue</title> +<para> +New packages might reside for some time in the so called new queue until +a member of the ftpmaster team has evaluated the package as fit for the +Debian distribution. To enable informing our users about that status the +new queue is parsed and the information displayed in the web sentinel pages. +The actual gatherer code can be found in UDD Git and is split into a +<ulink url="https://salsa.debian.org/qa/udd/blob/master/scripts/fetch_ftpnew.sh";>Script that fetches the data</ulink> +and a <ulink url="https://salsa.debian.org/qa/udd/blob/master/udd/ftpnew_gatherer.py";>Parser which injects the data into UDD</ulink>. +</para> + </sect2> + + <sect2 id="machine_readable"> + <title>Machine readable data in Git repositories of Blends and some packaging teams</title> +<para> +Several machine readable data of packages are parsed from Git repositories. +There are data that might be more up to date in Git (for instance scientific +publication data or some registry entries). There are also data for not +yet available packages a Blends team is working on. To enable refering to +this data some selected Git repositories are parsed. The parser for Salsa +is available in the +<ulink url="https://salsa.debian.org/blends-team/website/blob/master/misc/machine_readable/fetch-machine-readable_salsa.py";>website Git</ulink> +of the Blends project. It is running in a daily cron job on the host +<ulink url="http://blends.debian.net";>blends.debian.net</ulink> and +creates an <ulink url="http://blends.debian.net/_machine-readable/machine-readable.tar.xz";>archive</ulink> +with all the machine readable files found in the specified Git repositories. +</para> +<para> +This <ulink url="http://blends.debian.net/_machine-readable/machine-readable.tar.xz";>archive</ulink> +is read into UDD by an +<ulink url="https://salsa.debian.org/qa/udd/blob/master/udd/blends_prospective_gatherer.py";>UDD +importer script</ulink> in a cron job which is totally independent from the job +that collects the data from Salsa. Due to the disconnected jobs running on +different hosts there might be some undetermined delay between changing +some metadata in a Git repository and the time when the data are available +in UDD. +</para> + </sect2> + +</sect1> </appendix> View it on GitLab: https://salsa.debian.org/blends-team/blends/commit/3228dff27d0983422d3ba0556b42594066c3ed12 --- View it on GitLab: https://salsa.debian.org/blends-team/blends/commit/3228dff27d0983422d3ba0556b42594066c3ed12 You're receiving this email because of your account on salsa.debian.org.
_______________________________________________ Blends-commit mailing list Blends-commit@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/blends-commit
