Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10584 )
Change subject: IMPALA-7077. Add a configuration for the max number of
partitions to load
......................................................................
Patch Set 1:
(2 comments)
Todd, what are your thoughts on having a "per table" property that can
blacklist that particular table from loading in Impala? For example, something
like
alter table foo set tblproperties ("load-in-impala" = "false")
The issue with having a global max partitions is that
- It is a start up flag (requires a restart)
- Lets say if someone sets it to 100k, we can still have multiple tables with
90k partitions or something like that
- In many cases users would like to blacklist some specific problematic tables
(say with large incremental stats) and having just a partition limit may not
help
Vuk and I were discussing this offline today and I thought I'd bring this up in
this CR. It is unclear if both these flags can co-exist or we should just have
one of them. Thoughts?
http://gerrit.cloudera.org:8080/#/c/10584/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10584/1//COMMIT_MSG@21
PS1, Line 21: worth the complexity to try to
: somehow limit the addition of partitions during a DML
> The problem is that the catalog update happens at the end of the DML, after
I lean towards (2) but bubble up a meaningful error message for the DML query
and not the generic message that we are not able to load the table. I prefer
(2) because (1) leaves the table on HDFS in an inconsistent state.
http://gerrit.cloudera.org:8080/#/c/10584/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/10584/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2067
PS1, Line 2067: if (curPartitionCount + allHmsPartitionsToAdd.size() >
HdfsTable.MAX_PARTITIONS_PER_TABLE) {
> its not explicitly stated for java in the style guide, but given the limits
Yea, we use 90 chars for Java too. Gerrit defaults to 100.
--
To view, visit http://gerrit.cloudera.org:8080/10584
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife99a97a891ed14675303ea472abb2932a72cb51
Gerrit-Change-Number: 10584
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bhara...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <vercego...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Jun 2018 22:00:04 +0000
Gerrit-HasComments: Yes
