Some of our customers have 2 million TI rows, so for larger deploys I could see this being a problem. (They like to keep history around, and I don't blame them. It gives me warm and fuzzies too)
I don't think we have to support all DBs equally when it comes to "large" scale. if one DB has a clear advtange over others in some circumstances then that's a valid reason for us to say so in our docs I would think. If someone can find a way to improve mysql preformance for that case, then great, but I don't think that should mean we shouldn't improve it for postgres, unless doing so requires a massive hacking/special-casing of the code (which I don't think it would in this case) -ash > On 3 Dec 2019, at 12:47, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > Never used BRIN. I think in general DB of Airflow is rather small under > most circumstances (comparing to the databases I used to work with) - and I > think it could be a premature optimisation to switch type of index without > an actual evidence it is a problem of any sort and that we should use it. > Especially that it is Postgres-only - if find out that is worth fixing in > Postgres (because there is a serious performance improvement we can get) > then I think we must find a way how to make similar improvements in MySQL. > There are some existing mysql installations out there. > > J. > > On Tue, Dec 3, 2019 at 1:15 AM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > >> Hello, >> >> Has anyone tried to introduce BRIN indexes into the database schema? >> >> BRIN index is a great solution when we have a table in which the data is >> arranged chronologically by date or sequence number. It provides similar >> benefits to horizontal partitioning or sharding but without needing to >> explicitly declare partitions. BRIN-based indexes consume significantly >> less memory than B-tree indexes. It is supported by PostgreSQL >> >> In one project, where I have over 10 million records in one single table, >> rows looup took over 5 minutes. After introduce BRIN indexes, the time has >> dropped to less than 0.5 seconds. I personally love it. >> >> At first glance, this database structure perfectly matches our use case. >> >> Best regards, >> Kamil Breguła >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
