[
https://issues.apache.org/jira/browse/HUDI-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit updated HUDI-6445:
------------------------------
Description:
CI has been unstable for the past few weeks. we need to triage them and fix it.
UT-spark datasource module times out after 3 hours.
[https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc]
* Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to
50 secs or more and hence reaching 3 hours
{code:java}
2023-06-20T05:03:58.6566739Z 52.124
org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[2]
2023-06-20T05:03:58.6567324Z 49.446
org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[1]
2023-06-20T05:03:58.6568005Z 48.659
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6568471Z 47.799
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6569093Z 47.586
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6569503Z 41.208
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[2]
2023-06-20T05:03:58.6570090Z 41.034
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[4]
2023-06-20T05:03:58.6570501Z 40.225
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[3]
2023-06-20T05:03:58.6571231Z 39.853
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6574224Z 39.357
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[1]
2023-06-20T05:03:58.6575261Z 38.995
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6575765Z 38.846
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6576470Z 35.404
org.apache.hudi.functional.TestMORDataSourceWithBucketIndex
testCountWithBucketIndex {code}
TestHoodieDeltaStreamer.testUpsertsMORContinuousMode
and testAsyncClusteringServiceWithCompaction
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19]
TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21]
TestWriteMergeOnRead.testUpsert
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35]
TestWriteMergeOnReadWithCompact.testUpsert
TestWriteCopyOnWrite.testSubtaskFails
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30]
TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetection
https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18122/logs/28
was:
CI has been unstable for the past few weeks. we need to triage them and fix it.
UT-spark datasource module times out after 3 hours.
[https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc]
* Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to
50 secs or more and hence reaching 3 hours
{code:java}
2023-06-20T05:03:58.6566739Z 52.124
org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[2]
2023-06-20T05:03:58.6567324Z 49.446
org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[1]
2023-06-20T05:03:58.6568005Z 48.659
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6568471Z 47.799
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6569093Z 47.586
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6569503Z 41.208
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[2]
2023-06-20T05:03:58.6570090Z 41.034
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[4]
2023-06-20T05:03:58.6570501Z 40.225
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[3]
2023-06-20T05:03:58.6571231Z 39.853
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6574224Z 39.357
org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
HoodieRecordType,
String}[1]
2023-06-20T05:03:58.6575261Z 38.995
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6575765Z 38.846
org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6576470Z 35.404
org.apache.hudi.functional.TestMORDataSourceWithBucketIndex
testCountWithBucketIndex {code}
TestHoodieDeltaStreamer.testUpsertsMORContinuousMode
and testAsyncClusteringServiceWithCompaction
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19]
TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21]
TestWriteMergeOnRead.testUpsert
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35]
TestWriteMergeOnReadWithCompact.testUpsert
TestWriteCopyOnWrite.testSubtaskFails
[https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30]
> Fix CI stability Jun 26, 2023
> -----------------------------
>
> Key: HUDI-6445
> URL: https://issues.apache.org/jira/browse/HUDI-6445
> Project: Apache Hudi
> Issue Type: Test
> Components: tests-ci
> Reporter: sivabalan narayanan
> Priority: Major
> Labels: pull-request-available
>
> CI has been unstable for the past few weeks. we need to triage them and fix
> it.
>
>
> UT-spark datasource module times out after 3 hours.
> [https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc]
> * Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to
> 50 secs or more and hence reaching 3 hours
> {code:java}
> 2023-06-20T05:03:58.6566739Z 52.124
> org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
> testFailEarlyForIncrViewQue
> ryForNonExistingFiles{HoodieTableType}[2]
> 2023-06-20T05:03:58.6567324Z 49.446
> org.apache.hudi.functional.TestIncrementalReadWithFullTableScan
> testFailEarlyForIncrViewQue
> ryForNonExistingFiles{HoodieTableType}[1]
> 2023-06-20T05:03:58.6568005Z 48.659
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testMORDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[1]
> 2023-06-20T05:03:58.6568471Z 47.799
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testMORDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[3]
> 2023-06-20T05:03:58.6569093Z 47.586
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testMORDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[2]
> 2023-06-20T05:03:58.6569503Z 41.208
> org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
> HoodieRecordType,
> String}[2]
> 2023-06-20T05:03:58.6570090Z 41.034
> org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
> HoodieRecordType,
> String}[4]
> 2023-06-20T05:03:58.6570501Z 40.225
> org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
> HoodieRecordType,
> String}[3]
> 2023-06-20T05:03:58.6571231Z 39.853
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testCOWDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[1]
> 2023-06-20T05:03:58.6574224Z 39.357
> org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType,
> HoodieRecordType,
> String}[1]
> 2023-06-20T05:03:58.6575261Z 38.995
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testCOWDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[3]
> 2023-06-20T05:03:58.6575765Z 38.846
> org.apache.hudi.functional.cdc.TestCDCDataFrameSuite
> testCOWDataSourceWrite{HoodieCDCSupple
> mentalLoggingMode}[2]
> 2023-06-20T05:03:58.6576470Z 35.404
> org.apache.hudi.functional.TestMORDataSourceWithBucketIndex
> testCountWithBucketIndex {code}
>
> TestHoodieDeltaStreamer.testUpsertsMORContinuousMode
> and testAsyncClusteringServiceWithCompaction
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19]
>
> TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21]
>
> TestWriteMergeOnRead.testUpsert
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35]
>
> TestWriteMergeOnReadWithCompact.testUpsert
> TestWriteCopyOnWrite.testSubtaskFails
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18110/logs/30]
>
> TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetection
> https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18122/logs/28
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
