[
https://issues.apache.org/jira/browse/IMPALA-8434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891493#comment-16891493
]
Quanlong Huang commented on IMPALA-8434:
----------------------------------------
[~Xiaomeng Zhang], I think your solution make sense but I can't reproduce the
hanging of "invalidate metadata" at first. Finally, I figured out that if you
remove the codes for removing all versions of the tables and functions, the
"invalidate metadata" command do hang. So I guest your changes may be something
like this:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
index 13cb620..9ddb888 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
@@ -19,6 +19,7 @@ package org.apache.impala.catalog;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
+import java.util.List;
import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
@@ -390,11 +391,16 @@ public class ImpaladCatalog extends Catalog implements
FeCatalog {
newDb.setCatalogVersion(catalogVersion);
addDb(newDb);
if (existingDb != null) {
+ for (Table tbl : existingDb.getTables()) {
+ newDb.addTable(tbl);
+ }
+ for (List<Function> functionList :
existingDb.getAllFunctions().values()) {
+ for (Function func : functionList) {
+ newDb.addFunction(func);
+ }
+ }
CatalogObjectVersionSet.INSTANCE.updateVersions(
existingDb.getCatalogVersion(), catalogVersion);
- CatalogObjectVersionSet.INSTANCE.removeAll(existingDb.getTables());
- CatalogObjectVersionSet.INSTANCE.removeAll(
- existingDb.getFunctions(null, new PatternMatcher()));
} else {
CatalogObjectVersionSet.INSTANCE.addVersion(catalogVersion);
}
{code}
Actually, we shouldn't remove the codes calling
"CatalogObjectVersionSet.INSTANCE.removeAll". Because "newDb.addTable" and
"newDb.addFunction" will also add versions into the CatalogObjectVersionSet
which is a *multi-set*. Removing those three lines causes the old table
versions being double counted. So the minVersion_ of CatalogObjectVersionSet
won't be updated since their counters can't decrease to 0.
For "invalidate metadata", in the coordinator side, database updates are
processed prior to the table/function updates:
[https://github.com/apache/impala/blob/eb97c746d2309fcf78ff3b50751cd5e27101539a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L124-L129].
So even if we retain the old tables/functions of a db, they'll be updated
later in processing the table/function updates of this db. So I think it's no
problems for "invalidate metadata".
Just uploaded a patch for this: [https://gerrit.cloudera.org/c/13904/]
> Alter Db leads to functions missing unless run "refresh functions"
> -------------------------------------------------------------------
>
> Key: IMPALA-8434
> URL: https://issues.apache.org/jira/browse/IMPALA-8434
> Project: IMPALA
> Issue Type: Bug
> Reporter: Xiaomeng Zhang
> Assignee: Xiaomeng Zhang
> Priority: Critical
>
> I was testing on master branch. In a database with java and native functions.
> Run queries below, all functions are missing after alter db until run
> "refresh functions" in db.
> {code:java}
> [localhost:21000] xm> show functions;
> Query: show functions
> +-------------+------------------------+-------------+---------------+
> | return type | signature | binary type | is persistent |
> +-------------+------------------------+-------------+---------------+
> | STRING | add10impala(STRING) | JAVA | true |
> | STRING | add10udf(STRING) | JAVA | true |
> | INT | add2(INT, INT) | NATIVE | true |
> | INT | add2xm(INT, INT) | NATIVE | true |
> | INT | addtwomaster(INT, INT) | NATIVE | true |
> +-------------+------------------------+-------------+---------------+
> Fetched 5 row(s) in 0.01s
> [localhost:21000] xm> alter database xm set owner user impala218;
> Query: alter database xm set owner user impala218
> +-------------------+
> | summary |
> +-------------------+
> | Updated database. |
> +-------------------+
> Fetched 1 row(s) in 0.59s
> [localhost:21000] xm> show functions;
> Query: show functions
> Fetched 0 row(s) in 0.01s
> [localhost:21000] xm> refresh functions xm;
> Query: refresh functions xm
> Query submitted at: 2019-04-18 14:19:00 (Coordinator:
> http://xiaomeng-OptiPlex-9020:25000)
> Query progress can be monitored at:
> http://xiaomeng-OptiPlex-9020:25000/query_plan?query_id=fa40cdffde223550:df2a6cc000000000
> Fetched 0 row(s) in 0.08s
> [localhost:21000] xm> show functions;
> Query: show functions
> +-------------+------------------------+-------------+---------------+
> | return type | signature | binary type | is persistent |
> +-------------+------------------------+-------------+---------------+
> | STRING | add10impala(STRING) | JAVA | true |
> | STRING | add10udf(STRING) | JAVA | true |
> | INT | add2(INT, INT) | NATIVE | true |
> | INT | add2xm(INT, INT) | NATIVE | true |
> | INT | addtwomaster(INT, INT) | NATIVE | true |
> +-------------+------------------------+-------------+---------------+
> Fetched 5 row(s) in 0.00s
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
