[
https://issues.apache.org/jira/browse/IMPALA-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971201#comment-16971201
]
ASF subversion and git services commented on IMPALA-9136:
---------------------------------------------------------
Commit 0a082174c5717018971a9b2462306ca26494d6d0 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0a08217 ]
IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
The minimal catalog object version of valid catalog objects is used to
implement global invalidate metadata in legacy catalog mode. Coordinator
sends DDL RPC to catalogd for global invalidate metadata and gets the
expected min catalog version in the response. It's the version when
catalogd starts to reset the entire catalog, which means when the reset
is done, all valid catalog objects should be associated with a catalog
version larger than it. Coordinator will wait until its min catalog
version exceeds this value, which means it has processed all the updates
of the reset propagated from the catalogd via statestored. If SYNC_DDL
is set, the coordinator will also wait until other coordinators reach
the same statestore topic version with it, so they have also processed
the same updates and had the latest catalog after reset.
In local catalog mode, the coordinator does not cache all the metadata.
Instead, it caches them on-demand (based on query requests), and removes
them based on the Guava cache configurations (size or TTL) or explicit
invalidation from the catalog topic updates. So it's hard to track the
minimal catalog object version correctly.
This patch adds a new field (lastResetCatalogVersion) in TCatalog to
propagate the catalog version when catalogd starts to reset the entire
metadata. Each time when catalogd generates a new topic update, it will
generate a TCatalogObject of CATALOG type containing the state of the
catalog which includes this new field.
When coordinator receives a new value of lastResetCatalogVersion in a
topic update, it means catalogd has reset the entire catalog.
Coordinator will then clear its cache to remove all stale catalog
objects. It's possible that some fresh items being removed too. They
will be refetched on demand.
After the invalidation, there are no catalog object cached with catalog
version <= lastResetCatalogVersion. Because stale cache has been cleared
and all metadata from catalogd is newer than lastResetCatalogVersion. So
lastResetCatalogVersion + 1 is the lower bound (included) of min catalog
object version of a coordinator.
This patch also exposes the lower bound of catalog object version of via
a new metric "catalog.catalog-object-version-lower-bound" to ease
debugging.
IMPALA-9136 is also fixed in this patch.
Tests:
- Recover all existing tests that have been disabled due to this
missing feature
- Add custom cluster test for concurrent DDLs with INVALIDATE METADATA
- Run CORE tests
Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549
Reviewed-on: http://gerrit.cloudera.org:8080/14307
Reviewed-by: Vihang Karajgaonkar <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> IllegalStateException in calling Table.toThrift() without holding the table
> lock
> --------------------------------------------------------------------------------
>
> Key: IMPALA-9136
> URL: https://issues.apache.org/jira/browse/IMPALA-9136
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Priority: Major
>
> In CatalogServiceCatalog.addTableToCatalogDelta(), we are calling
> tbl.getUniqueName() without holding the table lock, which could finally
> results in an IllegalStateException:
> {code}
> I1107 17:34:27.058262 6381 CatalogServiceCatalog.java:1118] Table
> test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_14_part is skipping
> topic update 423
> I1107 17:34:27.058709 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_15_part,
> version=422, original size=2299, compressed size=1219
> I1107 17:34:27.061233 6381 jni-util.cc:288] java.lang.IllegalStateException:
> Table.toThrift() called without holding the table lock:
> test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part
> org.apache.impala.catalog.HdfsTable
> at org.apache.impala.catalog.Table.toThrift(Table.java:448)
> at org.apache.impala.catalog.HdfsTable.toThrift(HdfsTable.java:1408)
> at org.apache.impala.catalog.Table.setTCatalogObject(Table.java:487)
> at
> org.apache.impala.catalog.CatalogObjectImpl.toTCatalogObject(CatalogObjectImpl.java:51)
> at
> org.apache.impala.catalog.CatalogObjectImpl.getUniqueName(CatalogObjectImpl.java:45)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addTableToCatalogDelta(CatalogServiceCatalog.java:1084)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.addDatabaseToCatalogDelta(CatalogServiceCatalog.java:961)
> at
> org.apache.impala.catalog.CatalogServiceCatalog.getCatalogDelta(CatalogServiceCatalog.java:710)
> at
> org.apache.impala.service.JniCatalog.getCatalogDelta(JniCatalog.java:148)
> I1107 17:34:27.129712 7356 HdfsTable.java:934] Reloading metadata for all
> partition(s) of test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16
> (INSERT)
> I1107 17:34:27.132397 7356 HdfsTable.java:630] Loaded file and block
> metadata for test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16
> partitions:
> I1107 17:34:27.132424 7356 HdfsTable.java:965] Incrementally loaded table
> metadata for: test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16
> I1107 17:34:27.157740 6381 status.cc:126] IllegalStateException:
> Table.toThrift() called without holding the table lock:
> test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part
> org.apache.impala.catalog.HdfsTable
> @ 0x1c5ae50 impala::Status::Status()
> @ 0x24f7ad2 impala::JniUtil::GetJniExceptionMsg()
> @ 0x1c414af impala::JniCall::Call<>()
> @ 0x1c3fe27 impala::JniUtil::CallJniMethod<>()
> @ 0x1c3e06d impala::Catalog::GetCatalogDelta()
> @ 0x1c14344 impala::CatalogServer::GatherCatalogUpdatesThread()
> @ 0x1c2e14e boost::_mfi::mf0<>::operator()()
> @ 0x1c2da93 boost::_bi::list1<>::operator()<>()
> @ 0x1c2d4a6 boost::_bi::bind_t<>::operator()()
> @ 0x1c2cd91
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x1ffb6e9 boost::function0<>::operator()()
> @ 0x2573dea impala::Thread::SuperviseThread()
> @ 0x257c16e boost::_bi::list5<>::operator()<>()
> @ 0x257c092 boost::_bi::bind_t<>::operator()()
> @ 0x257c055 boost::detail::thread_data<>::run()
> @ 0x3d61599 thread_proxy
> @ 0x7f1ce6ca46b9 start_thread
> @ 0x7f1ce343f41c clone
> E1107 17:34:27.157758 6381 catalog-server.cc:379] IllegalStateException:
> Table.toThrift() called without holding the table lock:
> test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part
> org.apache.impala.catalog.HdfsTable
> I1107 17:34:27.158537 6389 catalog-server.cc:337] A catalog update with 4
> entries is assembled. Catalog version: 419 Last sent catalog version: 419
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
