[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel reassigned this task from Manuel to GoranSMilovanovic.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic, Manuel
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel closed this task as "Resolved".
Manuel claimed this task.
Manuel added a comment.


  Thank you! \o/

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel
  
  Following our 1:1
  
  - the criterion to keep the top 100 items per table is in force now in the 
test environment ;
  - the table loading times have significantly improved;
  - I am putting the system in production now.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel
  
  > Filters:
  > min 2 editors
  > max 100 lines per table from top
  
  Please be reminded that we already keep items with `min 2 editors`, and if we 
go for `max 100 lines per table` then we will end up filtering out items that 
indeed had `min 2 editors` but do not enter the table because of their row 
number in it merely.
  
  I remember when I started working on this dashboard and how I realised that 
the problem of what items to present deemed more complicated than what a pure 
intuition would suggest! Would you have some time for a very concise 1:1 on 
this, perhaps in the following two hours or so? Thank you. We can figure out 
the criteria we just need to be aware fully of the consequences of each 
(non-ideal) option that we have.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel added a comment.


  > all 6h should be included in 24h, all 24h should be included in 48h, and 
all 48h should be included in 72h
  
  This was the logic that I followed for debugging the tool, nothing more. I 
explained this only for you to understand why I think that the data shown made 
no sense at that time. This is not relevant beyond that.
  
  > if we keep the top 100 items per number of editors, then the top 100 in the 
72h dataset might encompass 
  > all items with at least, say, 7 editors, while in the 6h dataset at the 
same time we could have the situation 
  > that no items have even reached 7 editors yet;
  
  Yes, this is no problem.
  
  **Sorting:**
  
  - primary: number of editors (high numbers first)
  - secondary: number of edits (high numbers first)
  
  **Filters:**
  
  - min 2 editors
  - max 100 lines per table from top
  
  Please let me know if you need any more information.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel @Tobi_WMDE_SW
  
  To clarify a bit:
  
  - if we keep the top 100 items per number of editors, then the top 100 in the 
72h dataset might encompass all items with at least, say, 7 editors, while in 
the 6h dataset at the same time we could have the situation that no items have 
even reached 7 editors yet;
  - that would contradict the idea of: `... all 6h should be included in 24h, 
all 24h should be included in 48h, and all 48h should be included in 72h...`.
  
  We need to understand **exactly** what filtering do we want to implement. Let 
me know of your thoughts and I will implement whatever filtering you find to be 
the most suitable for this system.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a subscriber: Tobi_WMDE_SW.
GoranSMilovanovic added a comment.


  @Manuel @Tobi_WMDE_SW
  
  > The code for label lookup can live in the client JS or on the server.
  
  Who in the WMDE is our JS specialist so that I can get in touch with them? I 
don't do JS as you know, I am Data Scientist.
  
  > Loading times are a nightmare now.. can you test to cut off each table 
after the top 100 entries?
  
  But that would contradict yours:
  
  > Logic dictates that all 6h should be included in 24h, all 24h should be 
included in 48h, and all 48h should be included in 72h. This is still not the 
case, so something *must* be wrong here.
  
  from the ticket description.
  
  Namely, if you keep top 100 entities, well... Then the  entities from the 72h 
dataset - to take an example only - would not necessarily encompass the 
entities in the 6h dataset...
  
  I think the best way to proceed would be for you to formulate **exactly** the 
sort criteria that would fit our users' needs the best. To clarify:
  
  - in each of the following datasets: 6h, 24h, 48h, 72h, we have
  - the number of editors who edited a particular item
  - and the number of edits for those items;
  - we already keep all items that were edited by two or more editors.
  
  Please: exactly what filter/sorting do you have in mind?
  
  Besides this open question, everything else is ready for production.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: Tobi_WMDE_SW, GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, 
Akuckartz, Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel added a comment.


  > This would work for now, but the better alternative would be to implement 
the label lookup in the backend.
  
  Just what we discussed: The code for label lookup can live in the client JS 
or on the server. The latter could cache this info so that lookup would need no 
time on page load.
  
  But as I mentioned: Time is running out on this, so anything will do to 
finally get this operational.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel
  
  > Loading times are a nightmare now.. can you test to cut off each table 
after the top 100 entries?
  
  **To Do**
  
  > This would work for now, but the better alternative would be to implement 
the label lookup in the backend.
  
  Please elaborate on this idea,

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel added a comment.


  Yes, the error seems to be fixed now. The quality of the data has improved 
dramatically as a result! \o/
  
  > Now we need a new criterion to filter out items
  
  Loading times are a nightmare now.. can you test to cut off each table after 
the top 100 entries?
  
  > either by number of editors who work on them
  
  Yes, only that criterion is relevant!

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel To clarify and help you follow the developments here:
  
  I think the whole problem was related to filtering out items based on their 
revision frequencies ranks, and not absolute revision frequencies.
  
  The frequency revision ranks could have differed across the 72h, 48h, 24h, 
and 6h tables.
  
  Now we need a new criterion to filter out items (either by number of editors 
who work on them or by the absolute number of revisions they receive) in order 
to avoid serving too long item lists under Current Events.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel
  
  This is possibly fixed now in the test environment 
, but we need another 
decision now:
  
  - in order to fix this obvious and annoying bug I removed **all filtering** 
of items that have received edits by two or more editors recently;
  - in the test environment you will see that the system now again produces 
large tables (i.e. all items that have received edits by two or more editors 
recently are found there);
  - thus, we need a new criterion on what items to filter our (say: anything 
with less than N edits is filtered out, or something similar).
  
  Please advise.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  @Manuel
  
  - Found one possible (and most probable) cause of the anomaly;
  - fixed;
  - restarted the update engine in the test environment 
;
  - monitoring.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  The anomaly **is now observed** in the test environment 
: complete code evaluation 
of the dashboard's update engine will take place following the implementation 
of changes requested in T294983 .

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  > Logic dictates that all 6h should be included in 24h, all 24h should be 
included in 48h, and all 48h should be included in 72h. This is still not the 
case, so something *must* be wrong here.
  
  This anomaly is not present in the **restarted** test environment 
 (yet). Still monitoring; 
complete code evaluation of the dashboard's update engine will take place 
following the implementation of changes requested in T294983 
.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[GoranSMilovanovic]

GoranSMilovanovic added a comment.


  > Logic dictates that all 6h should be included in 24h, all 24h should be 
included in 48h, and all 48h should be included in 72h. This is still not the 
case, so something *must* be wrong here.
  
  I have restarted the update engine on the test server 
 and cleared all data. I 
will be monitoring the accumulation of data across the specified (6h, 24h, 48h, 
and 72h) intervals to find out why is this happening.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: GoranSMilovanovic
Cc: GoranSMilovanovic, Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, 
Nandana, Lahi, Gq86, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T296387: Some tables show wrong or incomplete information on Current Events dashboard

[Manuel]

Manuel created this task.
Manuel added projects: Wikidata Analytics, WMDE-Analytics-Engineering, 
User-GoranSMilovanovic, Wikidata.

TASK DESCRIPTION
  **Main components:**
  
  - Current Events Dashboard 
 (T259105 
)
  
  **Problem:**
  
  - Logic dictates that all 6h should be included in 24h, all 24h should be 
included in 48h, and all 48h should be included in 72h. This is still not the 
case, so something *must* be wrong here.
  
  **Acceptance criteria:**
  
  [ ] Error fixed.

TASK DETAIL
  https://phabricator.wikimedia.org/T296387

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Aklapper, Manuel, Invadibot, maantietaja, Akuckartz, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org