[
https://issues.apache.org/jira/browse/IMPALA-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326299#comment-17326299
]
Quanlong Huang commented on IMPALA-10663:
-----------------------------------------
I think this is not related to local catalog mode but an issue of the event
processor. I can't reproduce the issue if I keep the event processor disabled:
{code:java}
bin/start-impala-cluster.py --impalad_args --use_local_catalog=true
--catalogd_args --catalog_topic_mode=minimal{code}
The reason why we don't see this in the legacy catalog mode is that DDLs are
executed much faster in local catalog mode.
BTW, this seems a similar sympton with IMPALA-10502. [~vihangk1] has a patch
for it and it's under review: [https://gerrit.cloudera.org/c/17308/]. It
probably fixes this issue as well.
> Coordinator might observe stale metadata in local catalog mode
> --------------------------------------------------------------
>
> Key: IMPALA-10663
> URL: https://issues.apache.org/jira/browse/IMPALA-10663
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Priority: Major
>
> Coordinator might observe stale metadata in local catalog mode, even if the
> DDL operation was executed on the same coordinator.
> The issue is reproducible in the dev environment using local catalog mode:
> {noformat}
> bin/start-impala-cluster.py --impalad_args --use_local_catalog=true
> --catalogd_args --catalog_topic_mode=minimal --catalogd_args
> --hms_event_polling_interval_s=1
> {noformat}
> {noformat}
> CREATE TABLE doub (d double);
> {noformat}
> Renaming the table back and forth and querying from it reproduces the issue:
> bin/impala-shell.sh -q "alter table doub rename to doub_tmp; select * from
> doub_tmp; alter table doub_tmp rename to doub; select * from doub; alter
> table doub rename to doub_tmp; select * from doub_tmp;alter table doub_tmp
> rename to doub; select * from doub; alter table doub rename to doub_tmp;
> select * from doub_tmp;alter table doub_tmp rename to doub; select * from
> doub; alter table doub rename to doub_tmp; select * from doub_tmp;alter table
> doub_tmp rename to doub; select * from doub; alter table doub rename to
> doub_tmp; select * from doub_tmp;alter table doub_tmp rename to doub; select
> * from doub; alter table doub rename to doub_tmp; select * from
> doub_tmp;alter table doub_tmp rename to doub; select * from doub; alter table
> doub rename to doub_tmp; select * from doub_tmp;alter table doub_tmp rename
> to doub; select * from doub; alter table doub rename to doub_tmp; select *
> from doub_tmp;alter table doub_tmp rename to doub; select * from doub; alter
> table doub rename to doub_tmp; select * from doub_tmp;alter table doub_tmp
> rename to doub; select * from doub; alter table doub rename to doub_tmp;
> select * from doub_tmp;alter table doub_tmp rename to doub; select * from
> doub; alter table doub rename to doub_tmp; select * from doub_tmp;alter table
> doub_tmp rename to doub; select * from doub; alter table doub rename to
> doub_tmp; select * from doub_tmp;alter table doub_tmp rename to doub; select
> * from doub; alter table doub rename to doub_tmp; select * from
> doub_tmp;alter table doub_tmp rename to doub; select * from doub; alter table
> doub rename to doub_tmp; select * from doub_tmp;alter table doub_tmp rename
> to doub; select * from doub; alter table doub rename to doub_tmp; select *
> from doub_tmp;alter table doub_tmp rename to doub; select * from doub; alter
> table doub rename to doub_tmp; select * from doub_tmp;alter table doub_tmp
> rename to doub; select * from doub; alter table doub rename to doub_tmp;
> select * from doub_tmp;alter table doub_tmp rename to doub; select * from
> doub;"
> {noformat}
> ...
> ... then, after a while ...
> ...
> Query: alter table doub rename to doub_tmp
> +--------------------------+
> | summary |
> +--------------------------+
> | Renaming was successful. |
> +--------------------------+
> Fetched 1 row(s) in 0.02s
> Query: select * from doub_tmp
> Query submitted at: 2021-04-20 14:49:52 (Coordinator: http://localhost:25000)
> ERROR: AnalysisException: Could not resolve table reference: 'doub_tmp'
> Could not execute command: select * from doub_tmp
> {noformat}
> Then, if I execute the same SELECT a bit later:
> {noformat}
> $ bin/impala-shell.sh -q "select * from doub_tmp"
> Fetched 0 row(s) in 0.12s
> {noformat}
> I think if the DDL operation succeeds, then the following operations on the
> same coordinator should see its effect.
> The current behavior makes tests flaky. The workaround is to set
> sync_ddl=true, but it makes these operations very slow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
