[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore closed subtask T284538: effectively lower Item ID rate limit for bad 
Item creation requests as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  In T44362#7125078 , 
@Lydia_Pintscher wrote:
  
  > Thank you! It seems then we're still skipping somewhere around 1% of IDs if 
I didn't calculate wrong. That still seems too high to me. Will dig into it 
more then.
  
  One option we can still consider is “punishing” users for bad API requests, 
as sketched out in T272032#6834213 
. Right now we let bad API 
requests burn through 90 IDs per minute, which is better than not rate limiting 
ID generation at all, but still allows for a lot of skipped IDs, I think.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  In T44362#7095447 , 
@MisterSynergy wrote:
  
  > Continuation of the table above (numbers taken from the revision history of 
https://www.wikidata.org/wiki/User:MisterSynergy/itemstats):
  >
  > | **Week ending ...** | **Total skipped items** | **weekly increase** |
  > | 27 February 2021| 8542979   | 
|
  > | 6 March 2021| 8550632   | +7653   
|
  > | 13 March 2021   | 8596041   | +45409  
|
  > | 20 March 2021   | 8630017   | +33976  
|
  > | 27 March 2021   | 8652766   | +22749  
|
  > | 3 April 2021| 8666101   | +13335  
|
  > | 10 April 2021   | 8679152   | +13051  
|
  > | 17 April 2021   | 8688972   | +9820   
|
  > | 24 April 2021   | 8704612   | +15640  
|
  > | 1 May 2021  | 8714560   | +9948   
|
  > | 7 May 2021  | 8724841   | +10281  
|
  > | 15 May 2021 | 8735465   | +10624  
|
  > |
  >
  > Looks fine to me. Maybe something happened during two weeks in March, but 
not excessively so that we would need to worry.
  
  Thank you! It seems then we're still skipping somewhere around 1% of IDs if I 
didn't calculate wrong. That still seems too high to me. Will dig into it more 
then.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Bugreporter]

Bugreporter added a comment.


  > This would mean we could move back to fetching Ids in the same transaction 
as the main writes (so the id claim would also be rolled back on failure).
  > But as long as we are okay having some skipped IDs I would propose not 
touching it more now, they are after all, only numbers!
  
  I am strongly against doing so in one transaction. Previously this results in 
database lock timeout, and this may be exploited by anyone without being 
detected (i.e. a strong DoS potential).

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[MisterSynergy]

MisterSynergy added a comment.


  Continuation of the table above (numbers taken from the revision history of 
https://www.wikidata.org/wiki/User:MisterSynergy/itemstats):
  
  | **Week ending ...** | **Total skipped items** | **weekly increase** |
  | 27 February 2021| 8542979   | |
  | 6 March 2021| 8550632   | +7653   |
  | 13 March 2021   | 8596041   | +45409  |
  | 20 March 2021   | 8630017   | +33976  |
  | 27 March 2021   | 8652766   | +22749  |
  | 3 April 2021| 8666101   | +13335  |
  | 10 April 2021   | 8679152   | +13051  |
  | 17 April 2021   | 8688972   | +9820   |
  | 24 April 2021   | 8704612   | +15640  |
  | 1 May 2021  | 8714560   | +9948   |
  | 7 May 2021  | 8724841   | +10281  |
  | 15 May 2021 | 8735465   | +10624  |
  |
  
  Looks fine to me. Maybe something happened during two weeks in March, but not 
excessively so that we would need to worry.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  @MisterSynergy Could you have another look at the current situation after the 
recent changes we made?

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Invadibot, maantietaja, 
Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore added a comment.


  Any further changes to avoid skipped Qids would likely need some larger 
changes to some core Wikibase code or how ids are generated.
  
  Skipping probably started happening more once we fixed T194299: Lock wait 
timeout exceeded in SqlIdGenerator::generateNewId 

  
  We did have another idea for solving that issue back then in T194299#4712763 

  
  In T194299#4712763 , 
@Addshore wrote:
  
  > One idea that @daniel and I had for this was simply to spread the ItemId 
counter out over 2 or more rows.. having one row deal with an odd counter and 
another deal with an even counter.
  > This would mean that ids would no longer be guaranteed to be created in 
order, and the counters might drift (but we could put something in place to fix 
that?), but the issue of too much stuff trying to work on a single row would be 
solved.
  
  This would mean we could move back to fetching Ids in the same transaction as 
the main writes (so the id claim would also be rolled back on failure).
  But as long as we are okay having some skipped IDs I would propose not 
touching it more now, they are after all, only numbers!

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, maantietaja, Akuckartz, Nandana, 
lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, abian, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  Thanks a lot!

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, abian, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[MisterSynergy]

MisterSynergy added a comment.


  | **Week ending ...** | **Total skipped items** | **weekly increase** |
  | 5 December 2020 | 8303905   | +1256459  
|
  | 12 December 2020| 8351248   | +47343  |
  | 19 December 2020| 8380252   | +29004  |
  | 26 December 2020| 8420623   | +40371  |
  | 2 January 2021  | 8431286   | +10663  |
  | 9 January 2021  | 8459454   | +28168  |
  | 16 January 2021 | 8473979   | +14525  |
  | 23 January 2021 | 8487360   | +13381  |
  | 30 January 2021 | 8505173   | +17813  |
  | 6 February 2021 | 8514746   | +9573   |
  | 13 February 2021| 8524740   | +9994   |
  | 20 February 2021| 8535448   | +10708  |
  | 27 February 2021| 8542979   | +7531   |
  |
  
  This does not look overly concerning to me in the past weeks, and I do not 
see excessive streaks of skipped QIDs either. I have no further insight as to 
why these Q-IDs are being skipped, but I could provide lists with skipped QIDs 
if this helps.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, abian, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  With T272032  now done, could you 
get some new statistics @MisterSynergy?

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, abian, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher closed subtask T272032: Add rate limit for creating Item IDs as 
Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, abian, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore closed subtask T268625: [20h] Investigate the significant number of 
skipped Item IDs for newly created Wikidata items as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lea_Lacroix_WMDE]

Lea_Lacroix_WMDE added a comment.


  Thanks a lot for the update @MisterSynergy !

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lea_Lacroix_WMDE
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[MisterSynergy]

MisterSynergy added a comment.


  New numbers from the past week, in order to keep the momentum here:
  
  Between 28 Nov 1:42 AM and 5 Dec 1:42 AM, around 1.250.000 QIDs have been 
skipped and around 175.000 new items have been created. The skip ratio on 
average over the entire week was ~87.5%.
  
  Related activity started on Nov 29, around noon, and ended on Dec 3 in the 
evening. The skip rate during that time was constantly somewhere between 3 and 
4 QIDs per second.
  
  If anyone is interested, I would upload plots similar as previously here.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: Lucas_Werkmeister_WMDE, Addshore, MisterSynergy, Lea_Lacroix_WMDE, 
Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, 
Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, 
Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, 
ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, 
jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, 
ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, 
jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Addshore]

Addshore added a comment.


  So I looked at a small gap here:
  
  - https://www.wikidata.org/wiki/Q101579997 (created & deleted)
  - https://www.wikidata.org/wiki/Q101579998 (created @ 1:41)
  - https://www.wikidata.org/wiki/Q10157 (skipped)
  - https://www.wikidata.org/wiki/Q10158 (skipped)
  - https://www.wikidata.org/wiki/Q101580001 (skipped)
  - https://www.wikidata.org/wiki/Q101580002 (created @ 1:44)
  - https://www.wikidata.org/wiki/Q101580003 (skipped)
  - https://www.wikidata.org/wiki/Q101580004 (created @ 1:47)
  
  So the skips don't even appear to be during a time of mass creations?
  I think the best way to investigate this would probably to be to add some 
fairly verbose logging around the entity creation flows in it's own log channel?
  Perhaps:
  
  - When Special:NewItem starts the process of creating a new item?
  - When wbeditentity start the process of creating a new item?
  - When an ID is claimed in the IdGenerator
  
  This should then allow us to progress toward seeing exactly which requests 
end up generating an ID that end up not completing / saving an entity.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, 
ChongDae, Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, 
jeremyb, Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, ChongDae, 
Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, jeremyb, 
Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Bugreporter]

Bugreporter added a comment.


  Another idea is to allow user to reclaim Qids that are never used. Don't know 
how bad it is.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, ChongDae, 
Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, jeremyb, 
Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Lydia_Pintscher]

Lydia_Pintscher added a project: Wikidata-Campsite.
Lydia_Pintscher added a comment.


  Urgh. Yeah, this is not good. I thought we make major headway on this issue 
with T232620  but apparently, that's 
not enough. Skipping over a quarter is _bad_. Adding this to camp so we can 
look at some more of the subtickets @Bugreporter created.
  Thanks for creating the stats!

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, ChongDae, 
Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, jeremyb, 
Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[MisterSynergy]

MisterSynergy added a comment.


  Reminder that this is still a thing:
  
  I maintain a weekly updated page that is tracking the number of skipped item 
IDs (among other things) at 
https://www.wikidata.org/wiki/User:MisterSynergy/itemstats. During the past 
week (diff 
),
 the QID counter was increased by 586.750 from Q101579998 to Q102166748. 
However, 450.082 (76.7%) were skipped QIDs, and only around 130.000 QIDs are 
actual new items. This means that more than three out of four QIDs were skipped 
over the period of the past week; it also means that we have increased the 
number of skipped QIDs by almost 7% within only one week to more than 7 million 
meanwhile.
  
  I know that we do not run out of QIDs and this seems not actually a very 
serious issue, but in some sense a densly packed QID space still seems 
desirable. I thus suggest to start another investigation how this can happen 
and eventually be mitigated.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MisterSynergy
Cc: MisterSynergy, Lea_Lacroix_WMDE, Bugreporter, StudiesWorld, ChongDae, 
Wikidata-bugs, jeblad, Legoktm, Nemo_bis, Merl, Lydia_Pintscher, jeremyb, 
Stryn, daniel, Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, 
QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Bugreporter]

Bugreporter added subtasks: T264448: Entity ID should not be assigned for 
label/description conflict, T264449: Entity ID should not be assigned for 
invalid entity data, T264450: Entity ID should not be assigned if rate limit is 
hit, T264451: Entity ID should not be assigned if blocked by AbuseFilter.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, 
Nemo_bis, Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, 
darthmon_wmde, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T44362: skipped item IDs

[Bugreporter]

Bugreporter reopened this task as "Open".
Bugreporter added a comment.


  Reopen as a more general tracking task.

TASK DETAIL
  https://phabricator.wikimedia.org/T44362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bugreporter
Cc: Bugreporter, StudiesWorld, ChongDae, Wikidata-bugs, jeblad, Legoktm, 
Nemo_bis, Merl, Lydia_Pintscher, jeremyb, Stryn, daniel, Akuckartz, 
darthmon_wmde, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs