[
https://issues.apache.org/jira/browse/HUDI-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17460314#comment-17460314
]
Manoj Govindassamy commented on HUDI-3031:
------------------------------------------
{code:java}
manoj@ManojG:junit957946365060014697/testtable_MERGE_ON_READ $ ls -l
.hoodie/metadata/.hoodie | grep -v "crc"
total 216
-rw-r--r-- 1 manoj staff 2213 Dec 15 14:37 0000000000000.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37 0000000000000.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
0000000000000.deltacommit.requested
-rw-r--r-- 1 manoj staff 2300 Dec 15 14:37 20211022151606.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151606.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151606.deltacommit.requested
-rw-r--r-- 1 manoj staff 2347 Dec 15 14:37 20211022151612.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151612.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151612.deltacommit.requested
-rw-r--r-- 1 manoj staff 2393 Dec 15 14:37 20211022151618.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151618.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151618.deltacommit.requested
-rw-r--r-- 1 manoj staff 2438 Dec 15 14:37 20211022151623.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151623.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151623.deltacommit.requested
-rw-r--r-- 1 manoj staff 2483 Dec 15 14:37 20211022151628.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151628.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151628.deltacommit.requested
-rw-r--r-- 1 manoj staff 2528 Dec 15 14:37 20211022151633.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151633.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151633.deltacommit.requested
-rw-r--r-- 1 manoj staff 2573 Dec 15 14:37 20211022151639.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151639.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151639.deltacommit.requested
-rw-r--r-- 1 manoj staff 2618 Dec 15 14:37 20211022151643.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151643.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151643.deltacommit.requested
-rw-r--r-- 1 manoj staff 2667 Dec 15 14:37 20211022151644.deltacommit
-rw-r--r-- 1 manoj staff 1847 Dec 15 14:37
20211022151644.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151644.deltacommit.requested
-rw-r--r-- 1 manoj staff 2089 Dec 15 14:37 20211022151644001.commit
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151644001.compaction.inflight
-rw-r--r-- 1 manoj staff 1844 Dec 15 14:37
20211022151644001.compaction.requested
-rw-r--r-- 1 manoj staff 2284 Dec 15 14:37 20211022151651.deltacommit
-rw-r--r-- 1 manoj staff 1851 Dec 15 14:37
20211022151651.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:37
20211022151651.deltacommit.requested
-rw-r--r-- 1 manoj staff 2330 Dec 15 14:38 20211215143805791.deltacommit
-rw-r--r-- 1 manoj staff 1851 Dec 15 14:38
20211215143805791.deltacommit.inflight
-rw-r--r-- 1 manoj staff 0 Dec 15 14:38
20211215143805791.deltacommit.requested
drwxr-xr-x 2 manoj staff 64 Dec 15 14:37 archived
-rw-r--r-- 1 manoj staff 425 Dec 15 14:37 hoodie.properties {code}
> TestHoodieDeltaStreamerWithMultiWriter time out due to async services and
> writer deadlock
> -----------------------------------------------------------------------------------------
>
> Key: HUDI-3031
> URL: https://issues.apache.org/jira/browse/HUDI-3031
> Project: Apache Hudi
> Issue Type: Task
> Components: Writer Core
> Reporter: Manoj Govindassamy
> Assignee: Manoj Govindassamy
> Priority: Blocker
> Fix For: 0.11.0
>
>
> Off late, TestHoodieDeltaStreamerWithMultiWriter started consistently failing
> for the MOR table type. The test spins off few pool threads to do table
> ingestion via back filling along with async compaction and clustering. After
> the data ingestion is completed the test endlessly waits for the the
> following condition to pass.
>
> {code:java}
> // Condition for parallel ingestion job
> Function<Boolean, Boolean> conditionForRegularIngestion = (r) -> {
> if (tableType.equals(HoodieTableType.MERGE_ON_READ)) {
>
> TestHoodieDeltaStreamer.TestHelpers.assertAtleastNDeltaCommitsAfterCommit(3,
> lastSuccessfulCommit, tableBasePath, fs());
> } else {
>
> TestHoodieDeltaStreamer.TestHelpers.assertAtleastNCompactionCommitsAfterCommit(3,
> lastSuccessfulCommit, tableBasePath, fs());
> }
> TestHoodieDeltaStreamer.TestHelpers.assertRecordCount(totalRecords,
> tableBasePath + "/*/*.parquet", sqlContext());
> TestHoodieDeltaStreamer.TestHelpers.assertDistanceCount(totalRecords,
> tableBasePath + "/*/*.parquet", sqlContext());
> return true;
> }; {code}
> Issue 1: The compaction thread and the writer thread are in deadlock
> {code:java}
> "async_compact_thread" #188 prio=5 os_prio=31 tid=0x00007f8c26266800
> nid=0x15803 waiting for monitor entry [0x0000700009d3e000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hudi.client.transaction.TransactionManager.endTransaction(TransactionManager.java:70)
> - waiting to lock <0x00000006c353f528> (a
> org.apache.hudi.client.transaction.TransactionManager)
> at
> org.apache.hudi.client.SparkRDDWriteClient.completeCompaction(SparkRDDWriteClient.java:312)
> at
> org.apache.hudi.client.SparkRDDWriteClient.commitCompaction(SparkRDDWriteClient.java:294)
> at
> org.apache.hudi.client.HoodieSparkCompactor.compact(HoodieSparkCompactor.java:59)
> at
> org.apache.hudi.async.AsyncCompactService.lambda$null$1(AsyncCompactService.java:89)
> at
> org.apache.hudi.async.AsyncCompactService$$Lambda$612/2034420774.get(Unknown
> Source)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "pool-22-thread-1" #143 prio=5 os_prio=31 tid=0x00007f8c0b125800 nid=0x12603
> waiting on condition [0x0000700006fb7000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hudi.client.transaction.FileSystemBasedLockProviderTestClass.tryLock(FileSystemBasedLockProviderTestClass.java:80)
> at
> org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:68)
> at
> org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:64)
> - locked <0x00000006c353f528> (a
> org.apache.hudi.client.transaction.TransactionManager)
> at
> org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:193)
> at
> org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:125)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:536)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:308)
> {code}
> Issue 2: Even after fixing the above my replacing the
> hoodie.write.lock.provider with the local lock provider, the end condition of
> 3 DeltaCommitAfterLastCommit is not met and the test times out. This needs to
> be digged further.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
