There were two reasons for not enabling compression on data files. First of all, the way "chunking" in snapshot/restore API was implemented didn't allow simple implementation of compression on data files. Moreover, the data files are already compressed to a certain degree. In my tests I was getting about 20% compressions rates on index data with recent versions of Elasticsearch (we have a limitation that we can compress only one file at a time). So, difficulties with implementation together with limit benefits made this feature not very compelling. After recent refactoring of the storage code, it's now much easier to add this feature in if it makes sense. However, I am really curious how you've got such great compression rates. Which version of Elasticsearch were these indices created with? Did you upgrade from older version of elasticsearch recently and most of your data files are created with older versions of Lucene?
On Thursday, September 18, 2014 10:53:58 AM UTC-4, Domonkos Tomcsanyi wrote: > > thank you again, I'll keep an eye on this issue. > > Domonkos > > 2014.09.18. 16:38 keltezéssel, David Pilato írta: > > I don't know. I think this could happen in the future but unsure though. > May be Igor could answer this? > > Here is a related doc PR: > https://github.com/elasticsearch/elasticsearch/pull/7654 > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fpull%2F7654&sa=D&sntz=1&usg=AFQjCNFe2yi4QlvBIRxBw9tDL3IOdwzsIQ> > > > -- > *David Pilato* | *Technical Advocate* | *Elasticsearch.com* > @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr > <https://twitter.com/elasticsearchfr> > > > Le 18 septembre 2014 à 16:35:14, Tomcsányi, Domonkos ([email protected]) > a écrit: > > Thank you for the answer, but may I know why? What is the reason behind > this? > > thanks, > Domonkos > > 2014.09.18. 16:15 keltezéssel, David Pilato írta: > > Only metadata are compressed. > > > > -- > > David ;-) > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > >> Le 18 sept. 2014 à 15:36, "Tomcsányi, Domonkos" <[email protected]> > <[email protected]> a écrit : > >> > >> Hi Everyone, > >> > >> I just started experimenting with the cool snapshot feature of ES > (using ES 1.3.2 on Ubuntu 14.04) using curator. I created a new repository > on a mounted NFS storage, using only the default options (compression > turned on). I checked it using curl: > >> > >> user@myserver:~# curl -XGET ' > http://IP_ADDRESS:9200/_snapshot/logBack?pretty' > >> { > >> "logBack" : { > >> "type" : "fs", > >> "settings" : { > >> "compress" : "true", > >> "location" : "/es_snapshots" > >> } > >> } > >> } > >> > >> So, after that I used curator to create a snapshot of some older > indices. The process finished after some minutes, so I decided to have a > look at the files it created. It turned out that the snapshot's files take > up exactly as much space as the indices did originally while they were in > the cluster, so no compression happened at all. This is kind of a problem > for me, because I assumed that compression will greatly reduce the size of > the indices I put in a snapshot. So is there anything I'm doing wrong? > >> > >> Thank you, > >> Domonkos > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > >> To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/541ADFEF.7080800%40modit.hu > . > >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/541AED9F.4030007%40modit.hu > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local > > <https://groups.google.com/d/msgid/elasticsearch/etPan.541aee74.625558ec.b066%40MacBook-Air-de-David.local?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8712a76-6eed-4a48-8a1e-95062e2f1e7d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
