fgiunchedi added a comment.
In T294355#7527157 <https://phabricator.wikimedia.org/T294355#7527157>, @jcrespo wrote: > number of files are (within reason) a non-blocker for bacula, as files are packaged into volumes. It is true that each file is stored as a mysql record, but that should be able to scale until dozens of (US) billons, although it may be slow to recover when rebuilding metadata. > > Most limiting factor would be the overall size + backup frequency for capacity planning. We don't have a lot of temporal data backed up, so not sure if we could come up with a strategy that saves space (e.g. if data is immutable, we may want to avoid full backups every day). What is the file/directory structure? If data is below e.g. 100GB I would consider it "small" and not requiring optimization. > > The typical backup schedule is incrementals of a set of paths every day, differentials every fortnite, and fulls monthly- however it is highly customizable per job. Thank you that's helpful to know, my hunch is that we'd want every other week backups since this is mainly a safety measure. File structure is one file per metric for graphite, with the filesystem path mirroring the graphite path (e.g. `foo.bar.baz.value` will be `/foo/bar/baz/value.wsp` on the filesystem). All files are expected to be around ~100k (e.g. the `daily` top level directory I mentioned earlier is ~100k files and 11G in size). TASK DETAIL https://phabricator.wikimedia.org/T294355 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: fgiunchedi Cc: jcrespo, Manuel, Michael, Addshore, fgiunchedi, Aklapper, Lucas_Werkmeister_WMDE, Bongo-Cat, Invadibot, Devnull, LSobanski, maantietaja, lmata, Akuckartz, Nandana, Robin.guo, Lahi, Gq86, herron, GoranSMilovanovic, QZanden, Marostegui, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
