Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15692 )
Change subject: IMPALA-9616 ...................................................................... Patch Set 1: (8 comments) http://gerrit.cloudera.org:8080/#/c/15692/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15692/1//COMMIT_MSG@7 PS1, Line 7: IMPALA-9616 nit: can you briefly describe the change in the first line. This plays nicer with tools that only show the commit title. E.g. git log --pretty=oneline http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_compression_codec.xml File docs/topics/impala_spill_compression_codec.xml: PS1: These are startup parameters for Impala instead of query options, so don't fit in this part of the docs. The startup parameters are not all documented in as consistent a way, they are mostly mentioned in the various administration topics. I think these fit in impala_disk_space.xml, which discusses how to set -scratch_dirs (a related option). https://jenkins.impala.io/job/docs-build-ub1604/1604/artifact/Impala/docs/build/html/topics/impala_disk_space.html http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_compression_codec.xml@52 PS1, Line 52: allowed These are more the recommended values instead of the only allowed values (I read this as excluding other values). This supports the full range of values supported by the COMPRESSION_CODEC query option. I don't think admins should generally be setting it to other values, but it's probably good to at least document the possibilities. http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_compression_codec.xml@64 PS1, Line 64: <b>Default:</b> <codeph>ZSTD</codeph> The default is NONE http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_compression_codec.xml@67 PS1, Line 67: You can use this query option with the DISK_SPILL_PUNCH_HOLES query option. This is a must, i.e. If you set --disk_spill_compression_codec to a value other than NONE, you must set --disk_spill_punch_holes to true. http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_punch_holes.xml File docs/topics/impala_spill_punch_holes.xml: http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_punch_holes.xml@41 PS1, Line 41: To limit the amount of data spilled during pre-aggregation, you : enable hole punching support. This is not right, it's not related to the preaggregation change, maybe just remove? http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_punch_holes.xml@49 PS1, Line 49: <p>True allocates a range of the actual compressed size and punch holes Not sure the best way to explain what it does, since it's relatively low level. My attempt is something like: "When this option is enabled, the hole punching feature supported by many filesystems is used to reclaim space in scratch files during execution of a query that spills to disk. This results in lower scratch space requirements in many cases, especially when combined with disk spill compression. When this option is not enabled, scratch space is still recycled by a query, but less effectively in many cases." http://gerrit.cloudera.org:8080/#/c/15692/1/docs/topics/impala_spill_punch_holes.xml@52 PS1, Line 52: <b>Type:</b> <codeph>BOOLEAN</codeph> We should mention that the filesystems for the directories specified in -scratch_dirs must support hole punching if this option is enabled. It's the same for the data cache (see impala_data_cache.xml - that always requires hole punching support). -- To view, visit http://gerrit.cloudera.org:8080/15692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3694fe97d74697777a8d50288b406b8eca0aa9fb Gerrit-Change-Number: 15692 Gerrit-PatchSet: 1 Gerrit-Owner: Kristine Hahn <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Thu, 09 Apr 2020 04:12:02 +0000 Gerrit-HasComments: Yes
