Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd ...................................................................... Patch Set 3: > Patch Set 2: > > > > > It makes sense to me to use a faster compression algorithm than > > > > bzip. Does the disk usage change much for this table? Another > > > > option is ZSTD which is almost as fast to compress as LZ4 with > > a > > > > higher compression ratio. > > > > > > Checked locally, writing looks even a bit faster for zstd than > > for > > > lz4. > > > > > > sizes of store_sales (1 replica): > > > > > > It makes sense to me to use a faster compression algorithm than > > > > bzip. Does the disk usage change much for this table? Another > > > > option is ZSTD which is almost as fast to compress as LZ4 with > > a > > > > higher compression ratio. > > > > > > Checked locally, writing looks even a bit faster for zstd than > > for > > > lz4. > > > > > > sizes of store_sales (1 replica): > > > > bzip: 123 MB > > zstd: 186 MB > > lz4 : 343 MB > > > > lz4 is indeed much bigger > > I vote for using zstd instead > > Yeah, I've seen cases in the past where the IO is the bottleneck on these > machines with EBS. I think ZSTD is fast enough that IO to EBS becomes the > bottleneck, so the extra compression gives it an edge over LZ4. ZSTD sounds > great to me. re/IO bandwidth: - local runs should not be a problem, as most of the developers run NVMe drives on their workstations. Even a regular SATA SSD can usually read/write at ~500 GB/sec unless you're dumping tens of gigabytes. - EBS drives on our CI infrastructure are provisioned for 250 MB/sec bandwidth, which I found to be a cost-effective tradeoff between performance and extra cost (GP3 EBS volumes provide 125 GB/sec base bandwidth, which can be increased at extra cost). This can be increased as long as the extra bandwidth results in reduced build times, offsetting the extra cost by saving money on EC2 VM instance charges. -- To view, visit http://gerrit.cloudera.org:8080/24586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117 Gerrit-Change-Number: 24586 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Laszlo Gaal <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 16 Jul 2026 10:20:32 +0000 Gerrit-HasComments: No
