[
https://issues.apache.org/jira/browse/IMPALA-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989229#comment-16989229
]
ASF subversion and git services commented on IMPALA-9135:
---------------------------------------------------------
Commit f9a52ca0e0cdfd7c23a36273b557c6385827b71b in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f9a52ca ]
IMPALA-9211: Fix adding new table to stale db due to concurrent reset
When adding a new table to the catalog, we first get the db object from
dbCache and then add the table into it. When doing reset(), i.e. global
INVALIDATE METADATA, we replace the whole dbCache with a new one and
load db and table names from HMS. These two operations have race
conflicts so should both be protected by a exclusive lock, i.e. write
lock of version lock.
Currently, CatalogServiceCatalog.addTable() does not get the db object
within the write lock, which may get a stale db object and add new table
into it. This patch moves the operations into the protection of write
lock.
Tests:
- Ran test_concurrent_ddls.py without errors for CreateTable like
IMPALA-9135 found.
- Ran Core tests
Change-Id: I83d2e5f00eabe61a42c948ec1685ce29cdea1592
Reviewed-on: http://gerrit.cloudera.org:8080/14820
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
> ---------------------------------------------------------------
>
> Key: IMPALA-9135
> URL: https://issues.apache.org/jira/browse/IMPALA-9135
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> This can be revealed by tests/custom_cluster/test_concurrent_ddls.py added in
> [https://gerrit.cloudera.org/c/14307]
> If running with INVALIDATE METADATA concurrently, the DDLs may run out of
> attemps in CatalogServiceCatalog.waitForSyncDdlVersion() to wait for the
> target update being sent, no matter how large we increase the maxNumAttempts.
> The error logs:
> {code:java}
> E1107 17:34:25.092439 7353 CatalogServiceCatalog.java:2626] Couldn't
> retrieve the covering topic version for catalog objects. Updated objects:
> [TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_9_part
> version: 349], deleted objects: []
> I1107 17:34:25.093451 7353 jni-util.cc:288]
> org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog
> topic version for the SYNC_DDL operation after 5 attempts.The operation has
> been successfully executed but its effects may have not been broadcast to all
> the coordinators.
> at
> org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:2630)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:414)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
> I1107 17:34:25.142006 6389 catalog-server.cc:337] A catalog update with 2
> entries is assembled. Catalog version: 356 Last sent catalog version: 355
> I1107 17:34:25.142168 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_15_part,
> version=357, original size=101, compressed size=98
> I1107 17:34:25.142215 6381 catalog-server.cc:641] Collected update:
> 1:CATALOG_SERVICE_ID, version=357, original size=49, compressed size=52
> I1107 17:34:25.142287 7356 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 357. Time to identify topic
> version (msec): 19
> I1107 17:34:25.192239 6389 catalog-server.cc:337] A catalog update with 2
> entries is assembled. Catalog version: 357 Last sent catalog version: 356
> I1107 17:34:25.192428 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16_part,
> version=358, original size=101, compressed size=98
> I1107 17:34:25.192462 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_11_part,
> version=359, original size=101, compressed size=98
> I1107 17:34:25.192484 6381 catalog-server.cc:641] Collected update:
> 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part,
> version=360, original size=101, compressed size=98
> I1107 17:34:25.192535 6381 catalog-server.cc:641] Collected update:
> 1:CATALOG_SERVICE_ID, version=360, original size=49, compressed size=52
> I1107 17:34:25.192613 7355 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 13
> I1107 17:34:25.192695 7351 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 45
> I1107 17:34:25.192734 7350 CatalogServiceCatalog.java:2642] Operation using
> SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic
> version (msec): 29
> I1107 17:34:25.222911 7353 status.cc:126] CatalogException: Couldn't
> retrieve the catalog topic version for the SYNC_DDL operation after 5
> attempts.The operation has been successfully executed but its effects may
> have not been broadcast to all the coordinators.
> @ 0x1c5ae50 impala::Status::Status()
> @ 0x24f7ad2 impala::JniUtil::GetJniExceptionMsg()
> @ 0x1c41987 impala::JniCall::Call<>()
> @ 0x1c3fec9 impala::JniUtil::CallJniMethod<>()
> @ 0x1c3e0e6 impala::Catalog::ExecDdl()
> @ 0x1c1ed17 CatalogServiceThriftIf::ExecDdl()
> @ 0x1cb3047 impala::CatalogServiceProcessor::process_ExecDdl()
> @ 0x1cb2d95 impala::CatalogServiceProcessor::dispatchCall()
> @ 0x1c08d65 apache::thrift::TDispatchProcessor::process()
> @ 0x20e8a0d
> apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x20de040 impala::ThriftThread::RunRunnable()
> @ 0x20df766 boost::_mfi::mf2<>::operator()()
> @ 0x20df5fc boost::_bi::list3<>::operator()<>()
> @ 0x20df348 boost::_bi::bind_t<>::operator()()
> @ 0x20df25b
> boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x1ffb6e9 boost::function0<>::operator()()
> @ 0x2573dea impala::Thread::SuperviseThread()
> @ 0x257c16e boost::_bi::list5<>::operator()<>()
> @ 0x257c092 boost::_bi::bind_t<>::operator()()
> @ 0x257c055 boost::detail::thread_data<>::run()
> @ 0x3d61599 thread_proxy
> @ 0x7f1ce6ca46b9 start_thread
> @ 0x7f1ce343f41c clone
> E1107 17:34:25.222932 7353 catalog-server.cc:112] CatalogException: Couldn't
> retrieve the catalog topic version for the SYNC_DDL operation after 5
> attempts.The operation has been successfully executed but its effects may
> have not been broadcast to all the coordinators.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
