Jim Apple has posted comments on this change. Change subject: IMPALA-2840: Don't store table location in partition location ......................................................................
Patch Set 2: (2 comments) A couple of architecture/design comments. Leaving the others alone for a bit while we hash out how this should work in principle. http://gerrit.cloudera.org:8080/#/c/2355/2//COMMIT_MSG Commit Message: Line 9: For a table with location "ABC", most partitions will have locations : like "ABC/DEF=2". The "ABC" part of the location does not need to be : stored in Catalog for each partition; we can compress it down to one : bit in the common case. > I agree that right now this is the common case, but I am wondering if this Some other ways we could get compression: 1. The TODO below, where we compress column names. 2. Try to factor out common prefixes generally, perhaps using a PATRICIA-trie-like scheme. 3. Use a dictionary-based compression mechanism like LZ. I don't know if any are tunable for shortish strings. I think which one is best really depends on (a) how close users are to OOMing their catalog and (b) what their partitions look like (nested under table location in hdfs or not, many partitioning columns or not). http://gerrit.cloudera.org:8080/#/c/2355/2/fe/src/main/java/com/cloudera/impala/catalog/HdfsPartition.java File fe/src/main/java/com/cloudera/impala/catalog/HdfsPartition.java: Line 77: is_relative_ = (table != null) && (location != null) > is there a case for allowing absolute locations? i think both table and loc The base for absolute locations is for when a user calls "alter table t partition p set location 'l'". -- To view, visit http://gerrit.cloudera.org:8080/2355 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8c67b6ce0f83de2f5277a528a9ce67e47d638adb Gerrit-PatchSet: 2 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Jim Apple <[email protected]> Gerrit-Reviewer: Dimitris Tsirogiannis <[email protected]> Gerrit-Reviewer: Jim Apple <[email protected]> Gerrit-Reviewer: Marcel Kornacker <[email protected]> Gerrit-HasComments: Yes
