Shawn Chang created HUDI-9039:
---------------------------------
Summary: Unneeded lock contention
Key: HUDI-9039
URL: https://issues.apache.org/jira/browse/HUDI-9039
Project: Apache Hudi
Issue Type: Bug
Reporter: Shawn Chang
An EMR Hudi user reported that when upgrading from 0.8 to 0.15, with their
existing multi-writer setup, they saw many lock contentions like below that
they didn't have in 0.8.0:
{code:java}
org.apache.hudi.exception.HoodieLockException: Unable to acquire lock, lock
object null
at
org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:91)
at
org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:59)
at
org.apache.hudi.client.BaseHoodieWriteClient.doInitTable(BaseHoodieWriteClient.java:1248)
at
org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1291)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:154)
{code}
This is likely because of BaseHoodieWriteClient in Hudi tries upgrade and init
MDT for every write, even when upgrade is not needed or MDT is not enabled. See
LOC:
https://github.com/apache/hudi/blob/f9b26dad4fcbf9d026aa31ae10ffa51a0db37935/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java#L1302
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
