Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16602 )
Change subject: IMPALA-10243: ConcurrentModificationException during parallel INSERTs ...................................................................... IMPALA-10243: ConcurrentModificationException during parallel INSERTs Impala might throw a ConcurrentModificationException during a high load of INSERTs to the same table. The exception happens during thrift serialization of TUpdateCatalogResponse which have a reference to the metastore table. The serialization happens without a lock, so another thread might modify the metastore table object in the meantime. This can potentially happen in CatalogOpExecutor.updateCatalog() which updates the catalog version and unsets table column statistics. For some reason I only saw this error with local catalog. The problem is that in Table.toThrift() we set a reference to the metastore table object instead of deep copying it. So my fix is to deep copy the metastore table, this prevents concurrent modifications. Testing * added stress test 'test_insert_stress.py' Change-Id: Ie656925d764d5eb26c318703ca425529ecf7a3a3 Reviewed-on: http://gerrit.cloudera.org:8080/16602 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/Table.java A tests/stress/stress_util.py M tests/stress/test_acid_stress.py A tests/stress/test_insert_stress.py 4 files changed, 150 insertions(+), 23 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16602 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie656925d764d5eb26c318703ca425529ecf7a3a3 Gerrit-Change-Number: 16602 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
