Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4 ...................................................................... Patch Set 2: > > > It makes sense to me to use a faster compression algorithm than > > > bzip. Does the disk usage change much for this table? Another > > > option is ZSTD which is almost as fast to compress as LZ4 with > a > > > higher compression ratio. > > > > Checked locally, writing looks even a bit faster for zstd than > for > > lz4. > > > > sizes of store_sales (1 replica): > > > > It makes sense to me to use a faster compression algorithm than > > > bzip. Does the disk usage change much for this table? Another > > > option is ZSTD which is almost as fast to compress as LZ4 with > a > > > higher compression ratio. > > > > Checked locally, writing looks even a bit faster for zstd than > for > > lz4. > > > > sizes of store_sales (1 replica): > > bzip: 123 MB > zstd: 186 MB > lz4 : 343 MB > > lz4 is indeed much bigger > I vote for using zstd instead Yeah, I've seen cases in the past where the IO is the bottleneck on these machines with EBS. I think ZSTD is fast enough that IO to EBS becomes the bottleneck, so the extra compression gives it an edge over LZ4. ZSTD sounds great to me. -- To view, visit http://gerrit.cloudera.org:8080/24586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117 Gerrit-Change-Number: 24586 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Wed, 15 Jul 2026 21:44:19 +0000 Gerrit-HasComments: No
