[
https://issues.apache.org/jira/browse/IMPALA-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617561#comment-16617561
]
Antoni Ivanov edited comment on IMPALA-2913 at 9/17/18 2:00 PM:
----------------------------------------------------------------
This is happening very often for us. Sometimes one node would be stuck this
way for hours.
Usually a node is able to start if we restart catalogd/statestored (and then
try to start the node) - but if it is started some time after the
statestored/catalog had been started it is stucked in "Missing tables were not
received" state
The very least the 2 minutes timeout can be configurable perhaps ? At the
moment it is hard-coded
There are a few (4-5) large tables - with 20-50 thousand partitions
Our version is v2.8.0-cdh5.11.1
was (Author: tozka):
This is happening very often for us. Sometimes one node would be stuck this
way for hours.
Usually a node is able to start if we restart catalogd/statestored (and then
try to start the node) - but if it is started some time after the
statestored/catalog had been started it is stucked in "Missing tables were not
received" state
The very least the 2 minutes timeout can be configurable perhaps ? At the
moment it is hard-coded
Our version is v2.8.0-cdh5.11.1
> Impalad fails to load tables in test runs
> -----------------------------------------
>
> Key: IMPALA-2913
> URL: https://issues.apache.org/jira/browse/IMPALA-2913
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Affects Versions: Impala 2.5.0
> Reporter: Henry Robinson
> Assignee: Vuk Ercegovac
> Priority: Major
> Fix For: Impala 2.5.0
>
> Attachments: catalog.stacks.out, gdb.txt.zip, impalad.stacks.out
>
>
> Build: http://sandbox.jenkins.cloudera.com/job/impala-master-cdh5-trunk/1714/
> {code:title=impalad}
> I0130 10:12:40.874454 24955 Frontend.java:864] analyze query drop database if
> exists `test_parquet_list_encodings_793898492` cascade
> I0130 10:13:41.530505 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:13:41.532280 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:13:43.693908 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:13:43.697772 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:13:51.970826 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:13:51.971350 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:14:10.715591 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:14:10.718114 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> I0130 10:15:41.542430 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:15:41.544220 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:15:43.707329 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:15:43.711132 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:15:51.980530 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:15:51.981075 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:16:10.728305 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:16:10.730826 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> I0130 10:17:41.554452 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:17:41.556248 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:17:43.720675 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:17:43.724488 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:17:51.990448 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:17:51.990978 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:18:10.740888 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:18:10.743422 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> I0130 10:19:41.566427 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:19:41.568217 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:19:43.733989 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:19:43.737788 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:19:52.000059 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:19:52.000630 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:20:10.753553 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:20:10.756093 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> I0130 10:21:41.578438 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:21:41.580238 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:21:43.747339 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:21:43.751142 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:21:52.009795 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:21:52.010337 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:22:10.766197 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:22:10.768699 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> I0130 10:23:41.590198 24944 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:23:41.591982 24944 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer_demographics
> I0130 10:23:43.760777 24948 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:23:43.764662 24948 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.household_demographics, tpcds_parquet.customer
> I0130 10:23:52.019451 24928 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:23:52.020020 24928 Frontend.java:808] Requesting prioritized load of
> table(s): functional_avro_snap.alltypestiny
> I0130 10:24:10.778784 24925 Frontend.java:883] Missing tables were not
> received in 120000ms. Load request will be retried.
> I0130 10:24:10.781328 24925 Frontend.java:808] Requesting prioritized load of
> table(s): tpcds_parquet.customer, tpcds_parquet.promotion
> {code}
> ... and so on until 4 hours have elapsed and the build gets aborted.
> In the catalog:
> {code:title=catalogd}
> I0130 10:12:10.705793 24706 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:12:40.875004 29633 CatalogOpExecutor.java:1031] Dropping database
> test_parquet_list_encodings_793898492
> I0130 10:12:40.875083 29633 MetaStoreClientPool.java:55] Creating
> MetaStoreClient. Pool Size = 0
> I0130 10:12:40.875221 29633 HiveMetaStoreClient.java:376] Trying to connect
> to metastore with URI thrift://localhost:9083
> I0130 10:12:40.875553 29633 HiveMetaStoreClient.java:421] Opened a connection
> to metastore, current connections: 136
> I0130 10:12:40.876093 29633 HiveMetaStoreClient.java:473] Connected to
> metastore.
> I0130 10:13:41.532707 24701 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:13:43.698223 24709 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 1
> I0130 10:13:43.698299 24696 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:13:51.971761 24700 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:14:10.718595 24694 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 1
> I0130 10:14:10.718680 24707 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:15:41.544662 24708 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:15:43.711565 24699 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 1
> I0130 10:15:43.711632 24705 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> I0130 10:15:51.981508 24697 TableLoadingMgr.java:278] Loading next table.
> Remaining items in queue: 0
> {code}
> .... nothing obvious.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
