[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[Nuria]

Nuria added a comment.


  Given that retention is not on puppet is this a setting that is communicated 
to a new node when it joins the cluster by the leader of the partition or 
similar?

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata, Nuria
Cc: Nuria, JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, 
darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[dcausse]

dcausse added a comment.


  @JAllemandou I think that is an option as well, the thing is that is it is 
transitional to help to bootstrap a test of the full pipeline. In the end we 
won't be using jumbo and thus won't be able to rely on a 30days retention on 
main so hopefully we'll be able to reset the retention back to 7days once we're 
done with the test.
  To circumvent this particular problem (time to make the dumps available > 
retention) we could either:
  
  - send back the events that matter back to kafka and have higher retention 
like you suggest
  - create a dedicated job running on the analytics network to read the events 
stored in HDFS and figure out a way to make the resulting data available in 
kafka main

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, 
darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[JAllemandou]

JAllemandou added a comment.


  An idea: How about sending back to kafka the update stream and make THAT one 
retention higher?
  Moving retention to 30 days for revision-create will make a lot of data stay 
that wouldn't be necessary (about half of the data), while keeping only the 
updates should be enough.
  Just an idea :)

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JAllemandou
Cc: JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, 
darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-analytics) [2020-05-27T13:42:02Z]  
increased Kafka topic retention in jumbo-eqiad to 31 days for 
(eqiad|codfw).mediawiki.revision-create - T253753 


TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, darthmon_wmde, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

[Ottomata]

Ottomata added a comment.


  Did 31 days:
  
$ kafka topics --alter --topic eqiad.mediawiki.revision-create --config 
retention.ms=267840
$ kafka topics --alter --topic codfw.mediawiki.revision-create --config 
retention.ms=267840

$ kafka topics --describe --topic 
'^(eqiad|codfw).mediawiki.revision-create$'
Topic:codfw.mediawiki.revision-create   PartitionCount:1
ReplicationFactor:3 Configs:retention.ms=267840
Topic: codfw.mediawiki.revision-create  Partition: 0Leader: 1002
Replicas: 1002,1003,1004Isr: 1004,1002,1003
Topic:eqiad.mediawiki.revision-create   PartitionCount:1
ReplicationFactor:3 Configs:retention.ms=267840
Topic: eqiad.mediawiki.revision-create  Partition: 0Leader: 1003
Replicas: 1003,1002,1005Isr: 1002,1005,1003

TASK DETAIL
  https://phabricator.wikimedia.org/T253753

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, darthmon_wmde, Nandana, 
Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, JAllemandou, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs