[
https://issues.apache.org/jira/browse/IMPALA-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837371#comment-16837371
]
ASF subversion and git services commented on IMPALA-8486:
---------------------------------------------------------
Commit a2c5d953b0fa6d69aa34eccaa13fa9aacd31ad10 in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a2c5d95 ]
IMPALA-8121: part 2: use local catalog in containers
This enables "modern" catalog features including the
local catalog and HMS notification support in the
dockerised minicluster by default.
The flags can be overridden if needed.
Skip tests affected by these bugs:
* IMPALA-8486 (LibCache invalidations)
* IMPALA-8458 (alter column stats)
* IMPALA-7131 (data sources not supported)
* IMPALA-7538 (HDFS caching DDL not supported)
* IMPALA-8489 TestRecoverPartitions.test_post_invalidate fails with
IllegalStateException
* IMPALA-8459 (cannot drop Kudu table)
* IMPALA-7539 (insert permission checks)
Fix handling of table properties in _get_properties()
to avoid including properties from unrelated sections.
This caused problems becase of additional properties
added by metastore event processing.
Rewrite test_partition_ddl_predicates() to change file formats rather
than use HDFS caching DDL.
Update the various test_kudu_col* tests to not expect staleness of
Kudu metadata for catalog V2.
Fix IMPALA-8464 so that testMetaDataGetColumnComments() allows the
table comment to be present, which is the new behaviour. Add a
new end-to-end test test_get_tables() that tests the precise
behaviour for different catalog versions so as to not lose
coverage.
Change-Id: I900d4b718cca98bcf86d36a2e64c0b6a424a5b7c
Reviewed-on: http://gerrit.cloudera.org:8080/13226
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> test_udf_update_via_drop and test_udf_update_via_create fail on local catalog
> -----------------------------------------------------------------------------
>
> Key: IMPALA-8486
> URL: https://issues.apache.org/jira/browse/IMPALA-8486
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Todd Lipcon
> Priority: Critical
>
> {noformat}
> TestUdfTargeted.test_udf_update_via_drop[protocol: beeswax | exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1,
> 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> tests/query_test/test_udfs.py:541: in test_udf_update_via_drop
> self._run_query_all_impalads(exec_options, query_stmt, ["New UDF"])
> tests/query_test/test_udfs.py:52: in _run_query_all_impalads
> assert result.data == expected
> E assert ['Old UDF'] == ['New UDF']
> E At index 0 diff: 'Old UDF' != 'New UDF'
> E Full diff:
> E - ['Old UDF']
> E + ['New UDF']
> ----------------------------
> {noformat}
> The tests are checking that the local UDF caches on each impalad get
> invalidated by a drop/create of a function referencing the HDFS file
> containing the UDF. The test fails because the local catalog, unlike the
> regular catalog, doesn't invalidate LibCache entries upon receiving a catalog
> update.
> I looked at this for long enough to realise that the invalidation mechanism
> is fundamentally broken - it doesn't work with dedicated executors. It also
> creates a race between the statestore updates and queries referencing the
> UDFs - if the queries win the race, then they can incorrectly use the old
> version that should have been invalidated.
> I think this is a potentially problematic issue because old JAR/SO versions
> could persist in the cache indefinitely if old versions are overwritten in
> place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
