Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21115
to look at the new patch set (#2).
Change subject: WIP IMPALA-12831: Add lock for partition map in HdfsTable
......................................................................
WIP IMPALA-12831: Add lock for partition map in HdfsTable
HdfsTable.toMinimalTCatalogObject() is not always invoked with holding
the table lock, e.g. in invalidating a table, we could replace an
HdfsTable instance with an IncompleteTable instance. We then invoke
HdfsTable.toMinimalTCatalogObject() to get the removed catalog object.
However, the HdfsTable instance could be modified in the meantime by a
concurrent DDL/DML that would reload it, e.g. a REFRESH statement. This
causes HdfsTable.toMinimalTCatalogObject() failed by
ConcurrentModificationException on the column/partition list.
There are two parts in HdfsTable.toMinimalTCatalogObject() that could be
failed by concurrent modifications. One is adding the columns, the other
is adding the partitions. The columns are not used in the result
consumers so we can simply ignore them. To protect reading the partition
map, this patch adds a ReentrantReadWriteLock in HdfsTable for it.
Refactor some code paths to avoid holding the lock during external RPCs,
e.g. loading file metadata.
TODO:
HdfsTable.getPartitions() returns the map values externally. To protect
its usage without exposing the partitionMapLock outside HdfsTable, we
have to make sure it's only used in impalad or tests.
Currently, there are several code paths in catalogd that use it, e.g.
FeFsTable.Utils.getPartitionFromThriftPartitionSpec() which is used in
many catalogd code paths. As a workaround, a deep copy of the value list
are returned in getPartitions().
Tests
- Added e2e test
- TODO: run CORE tests
Change-Id: I7466c291e4619b1ceac325b84f1eadc598aef29e
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M tests/custom_cluster/test_concurrent_ddls.py
7 files changed, 360 insertions(+), 173 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/21115/2
--
To view, visit http://gerrit.cloudera.org:8080/21115
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7466c291e4619b1ceac325b84f1eadc598aef29e
Gerrit-Change-Number: 21115
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
