We are indexing all sort of events (Windows, Linux, Apache, Netflow and so on...) and impact is defined in speed of the Kibana GUI / how long it takes to load 7 or 14 days of data. Thats what is important for my colleagues.
Am Montag, 4. August 2014 10:52:25 UTC+2 schrieb Mark Walkom: > > What sort of data are you indexing? When you said performance impact was > minimal, how minimal and at what points are you seeing it? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] <javascript:> > web: www.campaignmonitor.com > > > On 4 August 2014 16:43, horst knete <[email protected] <javascript:>> > wrote: > >> Hi again, >> >> a quick report regarding compression: >> >> we are using a 3-TB btrfs-volume with 32k block size now which reduced >> the amount of data from 3,2 TB to 1,1TB without any segnificant performance >> losses ( we are using a 8 CPU, 20 GB Memory machine with an iSCSI.Link to >> the volume ). >> >> So for us i can only suggest using the btrfs-volume for long term storage. >> >> Am Montag, 21. Juli 2014 08:48:12 UTC+2 schrieb Patrick Proniewski: >>> >>> Hi, >>> >>> gzip/zlib compression is very bad for performance, so it can be >>> interesting for closed indices, but for live data I would not recommend it. >>> Also, you must know that: >>> >>> Compression using lz4 is already enabled into indices, >>> ES/Lucene/Java usually read&write 4k blocks, >>> >>> -> hence, compression is achieved on 4k blocks. If your filesystem uses >>> 4k blocks and you add FS compression, you will probably have a very small >>> gain, if any. I've tried on ZFS: >>> >>> Filesystem Size Used Avail Capacity Mounted on >>> zdata/ES-lz4 1.1T 1.9G 1.1T 0% /zdata/ES-lz4 >>> zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES >>> >>> If you are using a larger block size, like 128k, a compressed filesystem >>> does show some benefit: >>> >>> Filesystem Size Used Avail Capacity Mounted on >>> zdata/ES-lz4 1.1T 1.1G 1.1T 0% >>> /zdata/ES-lz4 -> compressratio 1.73x >>> zdata/ES-gzip 1.1T 901M 1.1T 0% >>> /zdata/ES-gzip -> compressratio 2.27x >>> zdata/ES 1.1T 1.9G 1.1T 0% /zdata/ES >>> >>> But a file system block larger than 4k is very suboptimal for IO (ES >>> read or write one 4k block -> your FS must read or write a 128k block). >>> >>> On 21 juil. 2014, at 07:58, horst knete <[email protected]> wrote: >>> >>> > Hey guys, >>> > >>> > we have mounted an btrfs file system with the compression method >>> "zlib" for >>> > testing purposes on our elasticsearchserver and copied one of the >>> indices >>> > on the btrfs volume, unfortunately it had no success and still got the >>> size >>> > of 50gb :/ >>> > >>> > I will further try it with other compression methods and will report >>> here >>> > >>> > Am Samstag, 19. Juli 2014 07:21:20 UTC+2 schrieb Otis Gospodnetic: >>> >> >>> >> Hi Horst, >>> >> >>> >> I wouldn't bother with this for the reasons Joerg mentioned, but >>> should >>> >> you try it anyway, I'd love to hear your findings/observations. >>> >> >>> >> Otis >>> >> -- >>> >> Performance Monitoring * Log Analytics * Search Analytics >>> >> Solr & Elasticsearch Support * http://sematext.com/ >>> >> >>> >> >>> >> >>> >> On Wednesday, July 16, 2014 6:56:36 AM UTC-4, horst knete wrote: >>> >>> >>> >>> Hey Guys, >>> >>> >>> >>> to save a lot of hard disk space, we are going to use an compression >>> file >>> >>> system, which allows us transparent compression for the es-indices. >>> (It >>> >>> seems like es-indices are very good compressable, got up to 65% >>> >>> compression-rate in some tests). >>> >>> >>> >>> Currently the indices are laying at a ext4-Linux Filesystem which >>> >>> unfortunately dont have the transparent compression ability. >>> >>> >>> >>> Anyone of you got experience with compression file systems like >>> BTRFS or >>> >>> ZFS/OpenZFS and can tell us if this led to big performance losses? >>> >>> >>> >>> Thanks for responding >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/1f9bf509-b185-4c66-99c5-d8f69e95bea8%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/1f9bf509-b185-4c66-99c5-d8f69e95bea8%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/211eebfe-768f-4ea9-a1c9-2c93b870e464%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
