akosiaris added a comment.
In T301471#7837692 <https://phabricator.wikimedia.org/T301471#7837692>, @Addshore wrote: > So a summary comment as promised! > > In T301471#7799393 <https://phabricator.wikimedia.org/T301471#7799393>, @Michaelcochez wrote: > >> @QChris I noticed the addition of the .gitreview file on gerrit. Is this file needed? If so, we would merge it into our github repository, so we can keep the active development there and synchronize with gerrit. > > Sounds like this would work to me, and would be fine from the WMDE side of things. > @akosiaris would syncing the code from Github to Gerrit via a Gerrit change whenever it would need to be updated be acceptable on your side? Yes, that would be ok, some other services use that approach too. > To summarize current state: > > - Repo was created, code is there https://gerrit.wikimedia.org/g/wikidata/propertysuggester/RecommenderServer/+/refs/heads/main but right now development still happens on github > - Blubber configuration exists in the repo, but who needs to setup the CI for this @akosiaris ? Is this on your team? or something we can do (we would need docs / guidence) Release Engineering would be the team for that. You can always send a gerrit change for integration/config (it should be pretty similar to 4bd7dc4ad5fb1 <https://phabricator.wikimedia.org/rCICF4bd7dc4ad5fb1d1c99a4bed049050e4bd26398ca>) and they can review/merge/deploy. > - bullseye distribution update was done as suggested above, does @Joe or @akosiaris want to review this PR? https://github.com/martaannaj/RecommenderServer/pull/22/files Done > - 'deployment-charts repository' was also mentioned, Again is there anything we can do on our side, or more something for service ops to handle? This is almost entirely self-serve. Docs are at: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Creating_a_Helm_Chart Once you have the chart from the above step, upload it to gerrit for review and we 'll work together there. > The other big topic is still how to include the index for the service. > Context, this index file is currently ~1.5GB and would need to be regenerated every now and again: > I see a couple of options, and I'm sure @akosiaris and team would have options. > > - Build 1 image with just the code and have another repo and another blubber config etc that then adds an index file in to a new image > - Where can this image file be stored and retrieved? is this a bit big to just shove in a gerrit repo? That's a large file. It's a bad idea to stick it in a docker registry, it will make deployments slower, more resource consuming and brittle. We are already meeting various limits (at multiple places) with large files and I can tell you that you don't want to be debugging them. Plus, conceptually, the place for a dataset that is used by a codebase, isn't right next to the codebase. To attack the problem from another angle. What is in that dataset? What is the format that is has and why ? Is it something we can alter? Would it make sense if it was stored in a datastore (e.g. SQL, Cassandra, etc) and queried instead ? It's also a bad idea to stick it in a git repo (any git repo). In fact, gerrit won't even allow you to push this. It has (and for good reasons) a `maxObjectSizeLimit` of 100MB. TASK DETAIL https://phabricator.wikimedia.org/T301471 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: akosiaris Cc: akosiaris, QChris, ItamarWMDE, Joe, Aklapper, Addshore, karapayneWMDE, Martaannaj, Michaelcochez, Astuthiodit_1, Arnoldokoth, Invadibot, maantietaja, wkandek, JMeybohm, Akuckartz, Nandana, jijiki, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Eevans, Hardikj, Wikidata-bugs, aude, Sjoerddebruin, Jdforrester-WMF, Mbch331, Jay8g, Dzahn
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
