Impala Public Jenkins has submitted this change and it was merged. ( )

Change subject: IMPALA-6215: Removes race when using LibCache.

IMPALA-6215: Removes race when using LibCache.

Re-do the previously reverted change for IMPALA-6215. This patch
addresses the flakes listed in IMPALA-6092, which have become more
urgent recently.

LibCache's api to provide access to locally cached files has a race.
Currently, the client of the cache accesses the locally cached path
as a string, but nothing guarantees that the associated file is not
removed before the client is done using it. This race is suspected
as the root cause for the flakiness seen in IMPALA-6092. These tests
fail once in a while with classloader errors unable to load java udf
classes. In these tests, the lib cache makes no guarantee that the path
to the jar will remain valid from the time the path is acquired through
the time needed to fetch the jar and resolve the needed classes.

LibCache offers liveness guarantees for shared objects via reference
counting. The fix in this patch extends this API to also cover paths
to locally cached files.

This fix *only* addresses the path race. General cleanup of the api will
be done separately.

   - added a test to that does many concurrent udf uses and
     removals. By increasing the concurrent operations to 100, the issue
     in IMPALA-6092 is locally reproducible on every run. With this fix,
     the problem is no longer reproducible with this test.

Change-Id: I72ac0dfb13cf37d79e25c5b8a258b65f2dad097f
Reviewed-by: Vuk Ercegovac <>
Tested-by: Impala Public Jenkins <>
M be/src/codegen/
M be/src/exec/
M be/src/exprs/
M be/src/exprs/hive-udf-call.h
M be/src/runtime/
M be/src/runtime/lib-cache.h
M be/src/service/
M tests/query_test/
8 files changed, 162 insertions(+), 50 deletions(-)

  Vuk Ercegovac: Looks good to me, approved
  Impala Public Jenkins: Verified

To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I72ac0dfb13cf37d79e25c5b8a258b65f2dad097f
Gerrit-Change-Number: 9968
Gerrit-PatchSet: 4
Gerrit-Owner: Vuk Ercegovac <>
Gerrit-Reviewer: Impala Public Jenkins <>
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-Reviewer: Vuk Ercegovac <>

Reply via email to