Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd ...................................................................... Patch Set 3: > > > > > It makes sense to me to use a faster compression algorithm > > than > > > > > bzip. Does the disk usage change much for this table? > Another > > > > > option is ZSTD which is almost as fast to compress as LZ4 > > with > > > a > > > > > higher compression ratio. > > > > > > > > Checked locally, writing looks even a bit faster for zstd > than > > > for > > > > lz4. > > > > > > > > sizes of store_sales (1 replica): > > > > > > > > It makes sense to me to use a faster compression algorithm > > than > > > > > bzip. Does the disk usage change much for this table? > Another > > > > > option is ZSTD which is almost as fast to compress as LZ4 > > with > > > a > > > > > higher compression ratio. > > > > > > > > Checked locally, writing looks even a bit faster for zstd > than > > > for > > > > lz4. > > > > > > > > sizes of store_sales (1 replica): > > > > > > bzip: 123 MB > > > zstd: 186 MB > > > lz4 : 343 MB > > > > > > lz4 is indeed much bigger > > > I vote for using zstd instead > > > > Yeah, I've seen cases in the past where the IO is the bottleneck > on > > these machines with EBS. I think ZSTD is fast enough that IO to > EBS > > becomes the bottleneck, so the extra compression gives it an edge > > over LZ4. ZSTD sounds great to me. > > realized that we actually never use non-parquet for tpcds > this code is misleading: > https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/tests/query_test/test_tpcds_queries.py#L49 > > due to workload_exploration_strategy mechanism get_workload() is > tpcds > > https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/bin/run-all-tests.sh#L189 I miss copypasted something. get_workload() is tpcds -> if get_workload() is tpccds, exploration_strategy() is always core -- To view, visit http://gerrit.cloudera.org:8080/24586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117 Gerrit-Change-Number: 24586 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Comment-Date: Thu, 16 Jul 2026 09:28:15 +0000 Gerrit-HasComments: No
