> On Jul 6, 2017, at 3:40 PM, Andreas Koenig > <andreas.koenig.7os6v...@franz.ak.mind.de> wrote: > >>>>>> On Thu, 6 Jul 2017 15:21:56 -0500, Doug Bell <d...@preaction.me> said: > >> The API I had everyone test was a backwards-compatibility API. We will >> gather data at multiple addresses, yes, but they all will feed to the >> same database. It's important that the fast-matrix stays operational, >> so the new API has a log.txt view same as the old API. The new log.txt >> reads from the shared database so that no matter how the data comes >> in, the fast-matrix will be able to pick it up. It's still the raw, >> unprocessed data, so any problems with backend processing will not >> hold up the fast-matrix data feed. (this is it, and it's slow, but it >> will be behind a Fastly cache and I'll do other things if necessary: >> http://metabase-beta.cpantesters.org/tail/log.txt) > > Just to make sure I did read that right. Are you saying > > [ ] fast matrix can continue to read > http://metabase.cpantesters.org/tail/log.txt > > or are you saying > > [ ] fast matrix needs to switch to > http://metabase-beta.cpantesters.org/tail/log.txt > <http://metabase-beta.cpantesters.org/tail/log.txt>
Sorry. When the new API is ready, I'll be moving it to metabase.cpantesters.org <http://metabase.cpantesters.org/>, so fast-matrix should not have to change at all. Fast matrix can continue to read http://metabase.cpantesters.org/tail/log.txt <http://metabase.cpantesters.org/tail/log.txt> (though, if you could make sure that http://metabase-beta.cpantesters.org/tail/log.txt <http://metabase-beta.cpantesters.org/tail/log.txt> is correctly parsed by fast-matrix, that would soothe any worries I have about the transition). > And do you have an idea why there might be the discrepancy of ~800000 > reports in fast-matrix and ~1M in matrix in the last 4 weeks? If > http://metabase.cpantesters.org/tail/log.txt is now significantly slower > than it was, that would explain it, it might indeed lead to data loss. Yeah, as Slaven mentioned, it got moved behind a Fastly cache when I fixed the SSL certificate problem. That also enabled a default cache of 1 hour, which ensured data loss: We get more than 1000 reports an hour (last I checked it was closer to 2000 reports per hour, and even looking at the log.txt I can see that there isn't an hour's worth of reports in it). I've updated the cache to be 2.5 minutes, which should fix that problem. If I can help at all with getting the data up-to-date, let me know.
