Michaelcochez added a comment.
Updated a patch to resolve the issues.
TASK DETAIL
https://phabricator.wikimedia.org/T287641
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Ottomata, Martaannaj, Michaelcochez, Aklapper, Invadibot
Michaelcochez added a subtask: T287797: request_duration_ms is -1 in A/B
testing when the request goes to the PropertySuggester.
TASK DETAIL
https://phabricator.wikimedia.org/T285098
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc
Michaelcochez added a comment.
With much appreciated help from @Ottomata , we got to see some events using
kafkacat on deployment-kafka-jumbo-2 now. To check whether the events are
correctly generated, this seems sufficient.
One thing which got done on the way is including the config
Michaelcochez added a subtask: T287641: Review wd_propertysuggester event
logging stream config.
TASK DETAIL
https://phabricator.wikimedia.org/T285098
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Lucas_Werkmeister_WMDE
Michaelcochez added a comment.
We (@Michaelcochez and @Martaannaj ) should have access now. We still have
to figure out where the events are logged. The `/srv/log/eventlogging`
directory were logging should happen according to TestingOnBetaCluster
<https://wikitech.wikimedia.org/w
Michaelcochez added a comment.
Turned out the instructions which we had found before are outdated (now
marked as deprecated). New ones are here:
https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#Deployment
Accordingly, I created a patch to get our streams activated
Michaelcochez added a subscriber: Ottomata.
Michaelcochez added a comment.
@Ottomata Is this a patch you can review?
TASK DETAIL
https://phabricator.wikimedia.org/T287641
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Ottomata
Michaelcochez closed this task as "Resolved".
Michaelcochez added a comment.
The example I was looking at fits the case you describe.
Server
{"meta":{"domain":"wikidata.beta.wmflabs.org","stream":"wd_propertysuggester.serv
Michaelcochez updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T285098
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Ottomata, Lucas_Werkmeister_WMDE, Martaannaj, Michaelcochez, Michael,
Addshore
Michaelcochez closed subtask T287797: request_duration_ms is -1 in A/B testing
when the request goes to the PropertySuggester as Resolved.
TASK DETAIL
https://phabricator.wikimedia.org/T285098
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
Michaelcochez added a comment.
For the security policy, I created a first draft. I would prefer to use
phabricator for any security issues. Smaller issues can go as issues on github.
https://github.com/martaannaj/RecommenderServer/blob/main/SECURITY.md
I have tried adding the CI
Michaelcochez added a comment.
@Aklapper : thanks. Process initiated.
https://phabricator.wikimedia.org/T296599
TASK DETAIL
https://phabricator.wikimedia.org/T292110
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Reedy, Michaelcochez
Cc
Michaelcochez added a comment.
We went forward fixing multiple aspects:
1. Linting, go-sec and running the tests is now part of the CI in the main
branch on github (@sbassett ) We fixed the flagged issues.
1. as part of this, we added several test cases back which we had removed
Michaelcochez added a comment.
@sbassett Is that something which should be checked now, during the security
readiness review, or only later upon deployment?
I have added the TLS option to the implementation, but the fact that we still
allow starting a http version remains flagged.
TASK
Michaelcochez closed this task as "Declined".
Michaelcochez added a subscriber: Martaannaj.
Michaelcochez added a comment.
Ok, I added myself and @Martaannaj to #mediawiki-extensions-propertysuggester
<https://phabricator.wikimedia.org/tag/mediawiki-extensions-propertysuggester/
Michaelcochez added a comment.
Thanks @sbassett . I didn't realize it was possible to run semgrep without
posting the results to their service. They actually have a configuration
available which can be used without that feature. I have now configured that.
Semgrep actually found
Michaelcochez added a comment.
I have now added gokart. The github action was not working out of the box,
because of some missing configuration parameters in the example. I opened a
pull request for that.
Then, I also added nancy to scan packages and enabled Dependabot alerts
Michaelcochez added a subtask: T286655: Review wd_propertysuggester A/B testing
schema.
TASK DETAIL
https://phabricator.wikimedia.org/T285098
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Lucas_Werkmeister_WMDE, Martaannaj
Michaelcochez added a comment.
@sbassett @Reedy thank you very much for your effort!
@Reedy could you have a look at the current security policy
https://github.com/martaannaj/RecommenderServer/security/policy and if this is
fine close https://github.com/martaannaj/RecommenderServer
Michaelcochez added a comment.
@Joe we have created files for blubber before, I assume what is needed is
very similar to that? I am not sure I know what the `deployment-charts
repository` is . The code is under development on
https://github.com/martaannaj/RecommenderServer/ also
Michaelcochez added a comment.
@Joe for the base image, would you recommend our current approach of starting
from an 'empty' image and downloading the latest go distribution ourselves, or
should we take this one: https://docker-registry.wikimedia.org/golang1.17/tags/
I am not sure how
Michaelcochez added a comment.
We'd be happy to receive a patch or pull request.
TASK DETAIL
https://phabricator.wikimedia.org/T301471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: ItamarWMDE, Joe, Aklapper, Addshore
Michaelcochez added a subscriber: QChris.
Michaelcochez added a comment.
@QChris I noticed the addition of the .gitreview file on gerrit. Is this file
needed? If so, we would merge it into our github repository, so we can keep the
active development there and synchronize with gerrit.
TASK
Michaelcochez added a comment.
@Joe : would it be fine if we include the index file into the image for the
A/B testing phase? If that is successful, we can see whether it makes sense to
move it to a different location.
We are fixing the blubber configuration in the coming days. Could
Michaelcochez added a comment.
@Joe We now made the changes to use the bullseye distribution and the
provided image with go installed See:
https://github.com/martaannaj/RecommenderServer/pull/22/files
@QChris We just went ahead and added that file in our repository.
This is now
Michaelcochez added a comment.
I merged the pull request on github now.
I do not have rights to push to the gerrit repository, it might just be my
limited knowledge of how gerrit works.
I will look into the helm chart/CI setup soon.
> questions around the index f
Michaelcochez added a comment.
@thcipriani Development is taking place here:
https://github.com/martaannaj/RecommenderServer
TASK DETAIL
https://phabricator.wikimedia.org/T344746
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc
Michaelcochez added a comment.
@thcipriani could you help me forward with this? If preferred, we could set
up a 20 minute call to get this all set and done.
TASK DETAIL
https://phabricator.wikimedia.org/T344746
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Michaelcochez added a comment.
@thcipriani I would like to take care of this. Since the version on gerrit is
pretty outdated by now, is it possible to directly work with the latest which
we develop on github?
I'd also be happy to move development completely to gitlab, if we can get our
CI
Michaelcochez added a comment.
An update on the current status, mainly regarding the index file:
First, I made a mistake in my response above. The size of the file is a lot
smaller than what I wrote above. The binary version is currently around 75mb
(and not 1.5gb).
Progress
Michaelcochez added a comment.
Regarding the option of using a batch of queries to an external database;
the issue is that what we are creating is a specialized index specifically for
what we need. What we perform is a tree traversal were at each node a new
decision is made. To do
Michaelcochez added a comment.
Do we have a timeline for this move already?
Is it better to not update the gerrit repo at the moment? Our main
development takes place on github and we have quite a few changes in the
pipeline.
TASK DETAIL
https://phabricator.wikimedia.org/T332953
EMAIL
Michaelcochez added a comment.
The testing code is now implemented, and we found two small issues with it.
These have now been resolved and the code is simplified further.
Give this ticket: https://phabricator.wikimedia.org/T332953 I am uncertain
whether it makes sense to merge things
Michaelcochez added a comment.
We do our development on github. Does it make more sense to restart with a
new repository on gitlab to mirror that, or better to migrate?
TASK DETAIL
https://phabricator.wikimedia.org/T332953
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
Michaelcochez added a subscriber: Martaannaj.
Michaelcochez updated the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T332953
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michaelcochez
Cc: Martaannaj, sbassett, bking, bd808
Michaelcochez added a comment.
Hi @akosiaris , apologies for the slow progress.
We have been updating several things, but I didn't follow up with the testing
code needed for the new parts. I have blocked time in the coming days to
proceed with this and then we can merge it in.
TASK
Michaelcochez added a comment.
@thcipriani Can you help me out? We recently updated to a new golang version,
and want to go forward getting this deployed properly.
TASK DETAIL
https://phabricator.wikimedia.org/T344746
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Michaelcochez added a comment.
I did quite an overhaul of the code here:
https://github.com/martaannaj/RecommenderServer to comply with the security and
deployment related issues. (see
https://phabricator.wikimedia.org/T292110#7613470 )
@Sjoerddebruin your status request urged me
38 matches
Mail list logo
