[
https://issues.apache.org/jira/browse/HUDI-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461718#comment-17461718
]
sivabalan narayanan edited comment on HUDI-3043 at 12/17/21, 11:13 PM:
-----------------------------------------------------------------------
One issue I found is the synchronized block in transaction manager. Async
compaction could not end the transaction, bcoz, a regular writer is waiting
inside transactionManager.beginTransaction and waiting for async compaction to
release the lock.
https://issues.apache.org/jira/browse/HUDI-3029
!Screen Shot 2021-12-17 at 3.07.39 PM.png!
Excerpt from the logs
{code:java}
28442 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Starting a txn for 20211217150833897, compaction
28443 [async_compact_thread] WARN
org.apache.hudi.client.transaction.TransactionManager - Latest completed
transaction instant Optional.empty
28457 [async_compact_thread] WARN
org.apache.hudi.client.transaction.TransactionManager - Transaction started
28457 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Starting txn complete for 20211217150833897, compaction
28488 [pool-23-thread-1] WARN org.apache.hudi.client.AbstractHoodieWriteClient
- Committing 20211217150834807 action deltacommit
28516 [pool-23-thread-1] WARN org.apache.hudi.client.AbstractHoodieWriteClient
- BBB Starting a txn for 20211217150834807, UPSERT
28516 [pool-23-thread-1] WARN
org.apache.hudi.client.transaction.TransactionManager - Latest completed
transaction instant Option{val=[20211217150830336__deltacommit__COMPLETED]}
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Starting a txn for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Starting txn complete for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Ending a txn for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Ending txn complete for 20211217150833897, UPSERT_PREPPED
29502 [pool-20-thread-1] WARN
org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - XXX going to
call shutdown gracefully for continuous job for batch222
29502 [pool-20-thread-1] WARN
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer - Shutting down
gracefully
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA Calling shutdown of async service with force value false
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA shutdown requested set to true
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService, calling shutdown of executor service explicitly
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA awaiting for executor service to shutdown after calling
shutdown
29994 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- Committing Compaction 20211217150833897. Finished with result
HoodieCommitMetadata{partitionToWriteStats={default=[HoodieWriteStat{fileId='621befc6-b796-44a3-bf20-a49c0d1e1dd7-0',
path='default/621befc6-b796-44a3-bf20-a49c0d1e1dd7-0_0-97-153_20211217150833897.parquet',
prevCommit='20211022151643', numWrites=3000, numDeletes=0,
numUpdateWrites=1660, totalWriteBytes=808618, totalWriteErrors=0,
tempPath='null', partitionPath='default', totalLogRecords=2000,
totalLogFilesCompacted=2, totalLogSizeCompacted=656580,
totalUpdatedRecordsCompacted=1660, totalLogBlocks=2, totalCorruptLogBlock=0,
totalRollbackBlocks=0}]}, compacted=true,
extraMetadata={schema={"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"timestamp","type":"long"},{"name":"_row_key","type":"string"},{"name":"partition_path","type":"string"},{"name":"rider","type":"string"},{"name":"driver","type":"string"},{"name":"begin_lat","type":"double"},{"name":"begin_lon","type":"double"},{"name":"end_lat","type":"double"},{"name":"end_lon","type":"double"},{"name":"distance_in_meters","type":"int"},{"name":"seconds_since_epoch","type":"long"},{"name":"weight","type":"float"},{"name":"nation","type":"bytes"},{"name":"current_date","type":{"type":"int","logicalType":"date"}},{"name":"current_ts","type":"long"},{"name":"height","type":{"type":"fixed","name":"fixed","namespace":"hoodie.source.hoodie_source.height","size":5,"logicalType":"decimal","precision":10,"scale":6}},{"name":"city_to_state","type":{"type":"map","values":"string"}},{"name":"fare","type":{"type":"record","name":"fare","namespace":"hoodie.source.hoodie_source","fields":[{"name":"amount","type":"double"},{"name":"currency","type":"string"}]}},{"name":"tip_history","type":{"type":"array","items":{"type":"record","name":"tip_history","namespace":"hoodie.source.hoodie_source","fields":[{"name":"amount","type":"double"},{"name":"currency","type":"string"}]}}},{"name":"_hoodie_is_deleted","type":"boolean"},{"name":"haversine_distance","type":"double"}]}},
operationType=UNKNOWN}
30007 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Ending a txn for 20211217150833897, compaction {code}
Check for commits "20211217150833897" and "20211217150834807".
was (Author: shivnarayan):
One issue is found is the synchronized block in transaction manager. Async
compaction could not end the transaction, bcoz, a regular writer is waiting
inside transactionManager.beginTransaction and waiting for async compaction to
release the lock.
https://issues.apache.org/jira/browse/HUDI-3029
!Screen Shot 2021-12-17 at 3.07.39 PM.png!
Excerpt from the logs
{code:java}
28442 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Starting a txn for 20211217150833897, compaction
28443 [async_compact_thread] WARN
org.apache.hudi.client.transaction.TransactionManager - Latest completed
transaction instant Optional.empty
28457 [async_compact_thread] WARN
org.apache.hudi.client.transaction.TransactionManager - Transaction started
28457 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Starting txn complete for 20211217150833897, compaction
28488 [pool-23-thread-1] WARN org.apache.hudi.client.AbstractHoodieWriteClient
- Committing 20211217150834807 action deltacommit
28516 [pool-23-thread-1] WARN org.apache.hudi.client.AbstractHoodieWriteClient
- BBB Starting a txn for 20211217150834807, UPSERT
28516 [pool-23-thread-1] WARN
org.apache.hudi.client.transaction.TransactionManager - Latest completed
transaction instant Option{val=[20211217150830336__deltacommit__COMPLETED]}
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Starting a txn for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Starting txn complete for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Ending a txn for 20211217150833897, UPSERT_PREPPED
28541 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- BBB Ending txn complete for 20211217150833897, UPSERT_PREPPED
29502 [pool-20-thread-1] WARN
org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - XXX going to
call shutdown gracefully for continuous job for batch222
29502 [pool-20-thread-1] WARN
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer - Shutting down
gracefully
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA Calling shutdown of async service with force value false
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA shutdown requested set to true
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService, calling shutdown of executor service explicitly
29503 [pool-20-thread-1] WARN org.apache.hudi.async.HoodieAsyncService -
DeltaSyncService AAA awaiting for executor service to shutdown after calling
shutdown
29994 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- Committing Compaction 20211217150833897. Finished with result
HoodieCommitMetadata{partitionToWriteStats={default=[HoodieWriteStat{fileId='621befc6-b796-44a3-bf20-a49c0d1e1dd7-0',
path='default/621befc6-b796-44a3-bf20-a49c0d1e1dd7-0_0-97-153_20211217150833897.parquet',
prevCommit='20211022151643', numWrites=3000, numDeletes=0,
numUpdateWrites=1660, totalWriteBytes=808618, totalWriteErrors=0,
tempPath='null', partitionPath='default', totalLogRecords=2000,
totalLogFilesCompacted=2, totalLogSizeCompacted=656580,
totalUpdatedRecordsCompacted=1660, totalLogBlocks=2, totalCorruptLogBlock=0,
totalRollbackBlocks=0}]}, compacted=true,
extraMetadata={schema={"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"timestamp","type":"long"},{"name":"_row_key","type":"string"},{"name":"partition_path","type":"string"},{"name":"rider","type":"string"},{"name":"driver","type":"string"},{"name":"begin_lat","type":"double"},{"name":"begin_lon","type":"double"},{"name":"end_lat","type":"double"},{"name":"end_lon","type":"double"},{"name":"distance_in_meters","type":"int"},{"name":"seconds_since_epoch","type":"long"},{"name":"weight","type":"float"},{"name":"nation","type":"bytes"},{"name":"current_date","type":{"type":"int","logicalType":"date"}},{"name":"current_ts","type":"long"},{"name":"height","type":{"type":"fixed","name":"fixed","namespace":"hoodie.source.hoodie_source.height","size":5,"logicalType":"decimal","precision":10,"scale":6}},{"name":"city_to_state","type":{"type":"map","values":"string"}},{"name":"fare","type":{"type":"record","name":"fare","namespace":"hoodie.source.hoodie_source","fields":[{"name":"amount","type":"double"},{"name":"currency","type":"string"}]}},{"name":"tip_history","type":{"type":"array","items":{"type":"record","name":"tip_history","namespace":"hoodie.source.hoodie_source","fields":[{"name":"amount","type":"double"},{"name":"currency","type":"string"}]}}},{"name":"_hoodie_is_deleted","type":"boolean"},{"name":"haversine_distance","type":"double"}]}},
operationType=UNKNOWN}
30007 [async_compact_thread] WARN org.apache.hudi.client.SparkRDDWriteClient
- false BBB Ending a txn for 20211217150833897, compaction {code}
Check for commits "20211217150833897" and "20211217150834807".
> Test failure with
> TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWriters
> -----------------------------------------------------------------------------------------------------
>
> Key: HUDI-3043
> URL: https://issues.apache.org/jira/browse/HUDI-3043
> Project: Apache Hudi
> Issue Type: Sub-task
> Affects Versions: 0.11.0
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2021-12-17 at 3.07.39 PM.png
>
>
> CI has been very flaky with this test.
> reference:
> [https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/4431/logs/35]
>
> Test fails after a timeout of 360 secs. This is what the test is doing.
> Instantiate two concurrent deltastreamer jobs. But are expected to conflict
> and one of them should fail.
>
> {code:java}
> 2021-12-15T12:31:08.1443043Z 415268 [pool-46-thread-1] WARN
> org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs`
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:08.1479662Z 415272 [pool-46-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error :
> 2021-12-15T12:31:08.1480755Z org.apache.hudi.exception.HoodieIOException:
> Could not load Hoodie properties from
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:08.1481566Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:08.1482274Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:08.1483350Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:08.1484084Z at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:08.1484912Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:08.1485842Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:08.1486735Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:08.1487473Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:08.1488117Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:08.1488795Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:08.1489389Z at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:08.1490080Z Caused by: java.io.FileNotFoundException: File
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
> does not exist
> 2021-12-15T12:31:08.1490870Z at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:08.1491603Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:08.1492301Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:08.1492988Z at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:08.1493700Z at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:08.1494397Z at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:08.1495022Z at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:08.1495676Z at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:08.1496385Z at
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:08.1497081Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:08.1497619Z ... 10 more
> 2021-12-15T12:31:09.0533739Z 416177 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
> 2021-12-15T12:31:09.0821375Z 416206 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
> 2021-12-15T12:31:09.4001348Z 06:50 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:09.4012877Z 06:50 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:09.4794927Z 06:50 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:11.4361520Z 06:52 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:12.3883215Z 419512 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
> 2021-12-15T12:31:12.4075450Z 419531 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
> 2021-12-15T12:31:12.7802513Z 06:54 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:12.7847857Z 06:54 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:12.8628212Z 06:54 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:13.1489569Z 420273 [pool-46-thread-1] WARN
> org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs`
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:13.1502918Z 420273 [pool-46-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error :
> 2021-12-15T12:31:13.1503992Z org.apache.hudi.exception.HoodieIOException:
> Could not load Hoodie properties from
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:13.1504809Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:13.1505526Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:13.1506253Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:13.1506983Z at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:13.1507927Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:13.1508853Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:13.1509879Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:13.1510645Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:13.1511272Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:13.1511948Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:13.1512551Z at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:13.1513607Z Caused by: java.io.FileNotFoundException: File
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
> does not exist
> 2021-12-15T12:31:13.1514412Z at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:13.1515141Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:13.1515860Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:13.1516555Z at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:13.1518333Z at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:13.1519089Z at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:13.1519729Z at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:13.1520387Z at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:13.1521094Z at
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:13.1521791Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:13.1522342Z ... 10 more
> 2021-12-15T12:31:14.7650788Z 06:56 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:15.5436369Z 422667 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
> 2021-12-15T12:31:15.5616963Z 422686 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
> 2021-12-15T12:31:15.8443523Z 06:57 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:15.8454420Z 06:57 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:15.9309839Z 06:57 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:17.8366064Z 06:59 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:18.1515156Z 425275 [pool-46-thread-1] WARN
> org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs`
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:18.1516556Z 425275 [pool-46-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error :
> 2021-12-15T12:31:18.1517464Z org.apache.hudi.exception.HoodieIOException:
> Could not load Hoodie properties from
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:18.1518265Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:18.1519344Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:18.1519949Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:18.1520520Z at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:18.1521158Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:18.1521875Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:18.1522917Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:18.1523479Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:18.1523916Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:18.1524404Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:18.1524820Z at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:18.1525313Z Caused by: java.io.FileNotFoundException: File
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
> does not exist
> 2021-12-15T12:31:18.1525956Z at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:18.1526494Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:18.1527013Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:18.1527495Z at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:18.1528014Z at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:18.1528523Z at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:18.1528952Z at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:18.1529418Z at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:18.1530054Z at
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:18.1530560Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:18.1530912Z ... 10 more
> 2021-12-15T12:31:18.6080652Z 425732 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
> 2021-12-15T12:31:18.6248321Z 425749 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
> 2021-12-15T12:31:18.9049323Z 07:00 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:18.9086840Z 07:00 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:18.9773610Z 07:00 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:20.8238902Z 07:02 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:21.8356036Z 428959 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
> 2021-12-15T12:31:21.8519931Z 428976 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
> 2021-12-15T12:31:22.1387636Z 07:03 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:22.1442159Z 07:03 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:22.1973498Z 07:03 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:23.1526405Z 430276 [pool-46-thread-1] WARN
> org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs`
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:23.1527490Z 430276 [pool-46-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error :
> 2021-12-15T12:31:23.1528146Z org.apache.hudi.exception.HoodieIOException:
> Could not load Hoodie properties from
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:23.1528881Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:23.1529399Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:23.1529934Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:23.1530477Z at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:23.1531100Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:23.1531842Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:23.1532547Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:23.1533092Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:23.1533647Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:23.1534137Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:23.1534554Z at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:23.1535044Z Caused by: java.io.FileNotFoundException: File
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
> does not exist
> 2021-12-15T12:31:23.1535642Z at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:23.1536186Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:23.1536689Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:23.1537183Z at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:23.1537702Z at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:23.1538216Z at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:23.1538644Z at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:23.1539108Z at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:23.1539625Z at
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:23.1540118Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:23.1540484Z ... 10 more
> 2021-12-15T12:31:24.0393154Z 07:05 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:24.2768054Z 431401 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:24.3011474Z 431423 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:24.5931674Z 07:05 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:24.5982776Z 07:05 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:24.6715314Z 07:06 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:26.4723566Z 07:07 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:26.6296934Z 433752 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
> 2021-12-15T12:31:26.6990277Z 433823 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
> 2021-12-15T12:31:26.7154263Z 433839 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
> 2021-12-15T12:31:27.0226015Z 07:08 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:27.0285268Z 07:08 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:27.0977866Z 07:08 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:28.1534844Z 435277 [pool-46-thread-1] WARN
> org.apache.hudi.common.table.HoodieTableConfig - Run `table recover-configs`
> if config update/delete failed midway. Falling back to backed up configs.
> 2021-12-15T12:31:28.1536264Z 435277 [pool-46-thread-1] WARN
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer - Got error :
> 2021-12-15T12:31:28.1537195Z org.apache.hudi.exception.HoodieIOException:
> Could not load Hoodie properties from
> /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
> 2021-12-15T12:31:28.1537984Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
> 2021-12-15T12:31:28.1539028Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
> 2021-12-15T12:31:28.1539640Z at
> org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
> 2021-12-15T12:31:28.1540183Z at
> org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
> 2021-12-15T12:31:28.1540885Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
> 2021-12-15T12:31:28.1541621Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
> 2021-12-15T12:31:28.1542337Z at
> org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
> 2021-12-15T12:31:28.1542884Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2021-12-15T12:31:28.1543318Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2021-12-15T12:31:28.1543805Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2021-12-15T12:31:28.1544221Z at java.lang.Thread.run(Thread.java:748)
> 2021-12-15T12:31:28.1544706Z Caused by: java.io.FileNotFoundException: File
> file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup
> does not exist
> 2021-12-15T12:31:28.1545306Z at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> 2021-12-15T12:31:28.1545840Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> 2021-12-15T12:31:28.1546345Z at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> 2021-12-15T12:31:28.1547105Z at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> 2021-12-15T12:31:28.1547635Z at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
> 2021-12-15T12:31:28.1548151Z at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
> 2021-12-15T12:31:28.1548578Z at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
> 2021-12-15T12:31:28.1549239Z at
> org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
> 2021-12-15T12:31:28.1549777Z at
> org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
> 2021-12-15T12:31:28.1550273Z at
> org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
> 2021-12-15T12:31:28.1550655Z ... 10 more
> 2021-12-15T12:31:28.8646121Z 07:10 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:29.0315217Z 436154 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
> 2021-12-15T12:31:29.0474647Z 436171 [main] WARN
> org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not
> found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
> 2021-12-15T12:31:29.9039470Z 07:10 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:29.9040844Z 07:10 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:29.9041727Z 07:10 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:31.2335559Z 07:12 WARN: Timeline-server-based markers are
> configured as the marker type but embedded timeline server is not enabled.
> Falling back to direct markers.
> 2021-12-15T12:31:32.1873178Z [ERROR] Tests run: 26, Failures: 0, Errors: 1,
> Skipped: 9, Time elapsed: 439.775 s <<< FAILURE! - in JUnit Vintage
> 2021-12-15T12:31:32.1874589Z [ERROR] HoodieTableType).[2]
> MERGE_ON_READ(testUpsertsContinuousModeWithMultipleWriters Time elapsed:
> 360.847 s <<< ERROR!
> 2021-12-15T12:31:32.1875409Z java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1876498Z Caused by: java.lang.RuntimeException:
> java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1877024Z Caused by: java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.1877203Z
> 2021-12-15T12:31:32.5675718Z [INFO]
> 2021-12-15T12:31:32.5710112Z [INFO] Results:
> 2021-12-15T12:31:32.5710772Z [INFO]
> 2021-12-15T12:31:32.5711020Z [ERROR] Errors:
> 2021-12-15T12:31:32.5711359Z [ERROR] java.lang.RuntimeException:
> java.util.concurrent.TimeoutException
> 2021-12-15T12:31:32.5711695Z [INFO]
> 2021-12-15T12:31:32.5712044Z [ERROR] Tests run: 26, Failures: 0, Errors: 1,
> Skipped: 9 {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
