fgiunchedi added a comment.
In T246004#6501106 <https://phabricator.wikimedia.org/T246004#6501106>, @Zbyszko wrote: > @fgiunchedi We estimate we'd need around 500GB of storage for the streaming updater (not accounting for replicas). Our use case is almost always write only (checkpoints are read only on pipeline restarts, which ideally will be done rarely) - but we have a elasticity when it comes to configuration of the checkpoints interval. 500G seems reasonable to me, do you have a sense how/if this size increases over time? (need to know in general but not a blocker ATM). Do you know if flink takes care of creating the containers too once it has access to swift? I'm asking because on container creation we can pick whether data will be stored on hdd or ssd. In T246004#6501794 <https://phabricator.wikimedia.org/T246004#6501794>, @Zbyszko wrote: > @fgiunchedi unfortunately, there is no docker on stat instances so I'm unable to test swift that way. I'd still prefer to have some container on a already running service (whichever is accessible from analytics cluster). Test we want to set up would involve longer running service, starting from longer intervals, going back to shorter ones. We want to set up full end to end solution, so it'd make sense to use a stable solution anyway. I understand where you are coming from and wanting to setup a full end to end solution. Where is (local?) testing of the Flink/WDQS pipeline happening at the moment? I'm asking because wherever that environment is then it'd be good to have a (even minimal, single host) Swift cluster to test the integration. TASK DETAIL https://phabricator.wikimedia.org/T246004 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, fgiunchedi Cc: elukey, EBernhardson, JMeybohm, fgiunchedi, CBogen, #analytics, dcausse, Gehel, Zbyszko, Aklapper, JAllemandou, Smalyshev, Iamamz3, Ottomata, NavinRizwi, Akuckartz, darthmon_wmde, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Dinoguy1000, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
