Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd ...................................................................... Patch Set 3: > > > > It makes sense to me to use a faster compression algorithm > than > > > > bzip. Does the disk usage change much for this table? Another > > > > option is ZSTD which is almost as fast to compress as LZ4 > with > > a > > > > higher compression ratio. > > > > > > Checked locally, writing looks even a bit faster for zstd than > > for > > > lz4. > > > > > > sizes of store_sales (1 replica): > > > > > > It makes sense to me to use a faster compression algorithm > than > > > > bzip. Does the disk usage change much for this table? Another > > > > option is ZSTD which is almost as fast to compress as LZ4 > with > > a > > > > higher compression ratio. > > > > > > Checked locally, writing looks even a bit faster for zstd than > > for > > > lz4. > > > > > > sizes of store_sales (1 replica): > > > > bzip: 123 MB > > zstd: 186 MB > > lz4 : 343 MB > > > > lz4 is indeed much bigger > > I vote for using zstd instead > > Yeah, I've seen cases in the past where the IO is the bottleneck on > these machines with EBS. I think ZSTD is fast enough that IO to EBS > becomes the bottleneck, so the extra compression gives it an edge > over LZ4. ZSTD sounds great to me. realized that we actually never use non-parquet for tpcds this code is misleading: https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/tests/query_test/test_tpcds_queries.py#L49 due to workload_exploration_strategy mechanism get_workload() is tpcds https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/bin/run-all-tests.sh#L189 -- To view, visit http://gerrit.cloudera.org:8080/24586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117 Gerrit-Change-Number: 24586 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 16 Jul 2026 09:26:04 +0000 Gerrit-HasComments: No
