[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a subscriber: gerritbot. gerritbot added a comment. Change 253571 had a related patch set uploaded (by Addshore): Social metrics to graphite https://gerrit.wikimedia.org/r/253571 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a comment. Change 253572 had a related patch set uploaded (by Addshore): Convert site_stats to graphite https://gerrit.wikimedia.org/r/253572 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a comment. Change 253573 had a related patch set uploaded (by Addshore): Convert getclaims stats to graphite https://gerrit.wikimedia.org/r/253573 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a comment. Change 253573 merged by Addshore: Convert getclaims stats to graphite https://gerrit.wikimedia.org/r/253573 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a comment. Change 253571 merged by jenkins-bot: Social metrics to graphite https://gerrit.wikimedia.org/r/253571 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
gerritbot added a comment. Change 253572 merged by Addshore: Convert site_stats to graphite https://gerrit.wikimedia.org/r/253572 TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, gerritbot Cc: gerritbot, fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
fgiunchedi added a subscriber: fgiunchedi. fgiunchedi added a comment. thanks for expanding on that, here's my (as the person who's been looking after our graphite stack) opinion: - graphite isn't really data warehouse, thus I wouldn't recommend it as the primary storage for the verbatim/authoritative data - though saving data in graphite for graphing/etc and archived elsewhere too I think would cater in this case - it is possible as @addshore suggests to not downsample daily data for a really long time, e.g. keeping a daily metric for e.g. 100y takes 438028 bytes on disk for each metric - an analytics graphite instance could help, it means maintenance of that too of course. - if the volume of metrics isn't very high (no idea on the order of magnitude though) then using the main graphite is certainly less overhead. To give an example, if we're talking about 10k distinct metrics that'd be no problem, 100k would be ATM. hope that helps! TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, fgiunchedi Cc: fgiunchedi, Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
Christopher added a subscriber: Christopher. Christopher added a comment. To expand on the use cases for a metrics storage backend here is appropriate. I think that Wikidata content metrics favor long term retention (i.e. forever) because their purpose is to evaluate dynamics over both short and long period intervals. Since content is always changing, recreation of a past state from live data is not possible. The value of these historical measurement "snapshots" is therefore quite high. These old data are never archived either and must be able to be retrieved without loading a dump or using some offline process. In contrast, ops metrics are much more focused on the present and/or recent state. Thus, two different use cases exist here. If the proposal to use Graphite can substantiate a long term ( not decaying ) storage method, then it should work for both. If not, then something else (like OpenTSDB/ HBase) should be implemented. TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, Christopher Cc: Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T117735: Track all Wikidata metrics currently gathered in Graphite rather than SQL and TSVs
Addshore added a comment. > If the proposal to use Graphite can substantiate a long term ( not decaying ) > storage method, then it should work for both Retention and resolution changes / decay are both configurable. Simply setting the retention to 1d:100y would / should keep daily metrics for a period of 100years TASK DETAIL https://phabricator.wikimedia.org/T117735 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Christopher, Aklapper, StudiesWorld, Addshore, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
