Bug#966297: UDD: mentors.debian.net schema change
Hi Lucas, The API is in production on mentors. I've worked a bit on udd and came up with a proposal: https://salsa.debian.org/qa/udd/-/merge_requests/29 Best, -- Baptiste BEAUPLAT - lyknode signature.asc Description: OpenPGP digital signature
Bug#966297: UDD: mentors.debian.net schema change
On 8/15/20 10:54 AM, Lucas Nussbaum wrote: > On 01/08/20 at 14:07 +0200, Baptiste BEAUPLAT wrote: >> On 8/1/20 9:01 AM, Lucas Nussbaum wrote: >>> Hi Baptiste, >>> >>> I think that it would be better if mentors.d.n would provide a JSON >>> export of its data that it useful for UDD. >> >> Would a JSON rest api work? > > Sure, provided it doesn't require looping over packages or uploads. Yeah. In this proposal [1], I've added a standalone /api/uploads/ route, listing all uploads with some additional hints (package name instead of just the id), in order for scripts not to have to fetch resources everywhere. [1]: https://salsa.debian.org/mentors.debian.net-team/debexpo/-/merge_requests/146 -- Baptiste BEAUPLAT - lyknode signature.asc Description: OpenPGP digital signature
Bug#966297: UDD: mentors.debian.net schema change
On 01/08/20 at 14:07 +0200, Baptiste BEAUPLAT wrote: > On 8/1/20 9:01 AM, Lucas Nussbaum wrote: > > Hi Baptiste, > > > > I think that it would be better if mentors.d.n would provide a JSON > > export of its data that it useful for UDD. > > Would a JSON rest api work? Sure, provided it doesn't require looping over packages or uploads. Lucas signature.asc Description: PGP signature
Bug#966297: UDD: mentors.debian.net schema change
On 8/1/20 9:01 AM, Lucas Nussbaum wrote: > Hi Baptiste, > > I think that it would be better if mentors.d.n would provide a JSON > export of its data that it useful for UDD. Would a JSON rest api work? UDD could fetch something like https://mentors.debian.net/api/uploads/ and retrieve all information needed. > Regarding data that should be replicated in UDD, I would prefer > something simpler that is enough, typically for DMD, to notify that > there's a new version waiting, but that still requires going to > mentors.d.n for details. I wonder if we should have more than one table, > that just describes the last upload for a given source or (source,user) > (it's not clear to me if several users can simultaneously upload > competing versions of the same source package). Yes, several users can upload the same package using different version and distributions. From what I can see in the DMD page, it fetches from UDD's tables: - package name - version - distribution - uploaded date From that, distribution is actually unused. So fetching the list of all uploads and storing that into a single table sould be sufficient. -- Baptiste BEAUPLAT - lyknode signature.asc Description: OpenPGP digital signature
Bug#966297: UDD: mentors.debian.net schema change
Hi Baptiste, On 26/07/20 at 10:39 +0200, Baptiste BEAUPLAT wrote: > Package: qa.debian.org > User: qa.debian@packages.debian.org > Usertags: udd > > Hi, > > mentors.debian.net has changed its framework and its database schema. > The UDD importer named `mentors_gatherer.rb`[1] is not compatible with > this new schema and will stop working. > > See the new proposal for the mentors UDD tables (attached to this mail): > > - mentors-core-schema.dot: The core schemas of mentors with all data > tables (without internal django stuff) > - mentors-useful-table-schema.dot: A proposal of tables that could be > used by the UDD gatherer to import data from > - mentors-udd-schema.dot: A proposal of tables that could be used by the > UDD gatherer to import data to > > Once we agree on the new schema for UDD for the mentors tables, I'll be > happy to provide a patch for the gatherer and the schema. > > Note that currently the password of udd postgresql mentors account lives > in cleartext, in the gatherer, in the repository. Thus, I'm disabling > immediately that access. I'll provide the new password but that will > have to stay out of the repo. > > [1]: https://salsa.debian.org/qa/udd/-/blob/master/udd/mentors_gatherer.rb I think that it would be better if mentors.d.n would provide a JSON export of its data that it useful for UDD. Regarding data that should be replicated in UDD, I would prefer something simpler that is enough, typically for DMD, to notify that there's a new version waiting, but that still requires going to mentors.d.n for details. I wonder if we should have more than one table, that just describes the last upload for a given source or (source,user) (it's not clear to me if several users can simultaneously upload competing versions of the same source package). Lucas signature.asc Description: PGP signature
