[
https://issues.apache.org/jira/browse/IMPALA-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mihaly Szjatinya updated IMPALA-15254:
--------------------------------------
Description:
`prepare_encoded_test_table_compress()` in `tests/query_test/test_charcodec.py`
creates a table and loads data via Hive (`run_stmt_in_hive`) but never calls
`INVALIDATE METADATA` or `REFRESH` afterward. Normally Impala picks up
Hive-created tables automatically via the HMS event processor. However, when
the event processor is lagging or has crashed (e.g. during long exhaustive runs
on resource-constrained environments), the asynchronous propagation may not
complete before `_compare_tables` queries the table, causing:
{code:java}
AnalysisException: Could not resolve table reference: '<db>.<charset>_gen_None'
{code}
*Fix:* Add {{INVALIDATE METADATA
{db}
.
{encoded_table}
}} (or {{{}REFRESH{}}}) at the end of `prepare_encoded_test_table_compress`,
consistent with the other helpers.
was:
`prepare_encoded_test_table_compress()` in `tests/query_test/test_charcodec.py`
creates a table and loads data via Hive (`run_stmt_in_hive`) but never calls
`INVALIDATE METADATA` or `REFRESH` afterward. Normally Impala picks up
Hive-created tables automatically via the HMS event processor. However, when
the event processor is lagging or has crashed (e.g. during long exhaustive runs
on resource-constrained environments), the asynchronous propagation may not
complete before `_compare_tables` queries the table, causing:
{code}
AnalysisException: Could not resolve table reference: '<db>.<charset>_gen_None'
{code}
All other {{prepare_*}} helpers in the same file explicitly sync the catalog
after Hive writes:
- {{prepare_utf8_test_table}} -- calls REFRESH (line 117)
- {{prepare_encoded_test_table}} -- calls REFRESH (line 129)
- {{prepare_utf8_test_table_partitions}} -- calls REFRESH (line 223)
- {{prepare_encoded_test_table_partitions}} -- calls REFRESH (line 240)
{{prepare_encoded_test_table_compress}} (line 173) is the only one that omits
this step. This is a test-reliability bug, not a product bug.
*Fix:* Add {{INVALIDATE METADATA {db}.{encoded_table}}} (or {{REFRESH}}) at the
end of `prepare_encoded_test_table_compress`, consistent with the other helpers.
> test_enc_dec_gen_compress in test_charcodec.py fails with AnalysisException
> when HMS event processor is slow
> ------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15254
> URL: https://issues.apache.org/jira/browse/IMPALA-15254
> Project: IMPALA
> Issue Type: Bug
> Reporter: Mihaly Szjatinya
> Assignee: Mihaly Szjatinya
> Priority: Major
> Labels: test-failure
>
> `prepare_encoded_test_table_compress()` in
> `tests/query_test/test_charcodec.py` creates a table and loads data via Hive
> (`run_stmt_in_hive`) but never calls `INVALIDATE METADATA` or `REFRESH`
> afterward. Normally Impala picks up Hive-created tables automatically via the
> HMS event processor. However, when the event processor is lagging or has
> crashed (e.g. during long exhaustive runs on resource-constrained
> environments), the asynchronous propagation may not complete before
> `_compare_tables` queries the table, causing:
> {code:java}
> AnalysisException: Could not resolve table reference:
> '<db>.<charset>_gen_None'
> {code}
> *Fix:* Add {{INVALIDATE METADATA
> {db}
> .
> {encoded_table}
> }} (or {{{}REFRESH{}}}) at the end of `prepare_encoded_test_table_compress`,
> consistent with the other helpers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
