yihua opened a new pull request #3921:
URL: https://github.com/apache/hudi/pull/3921
Reverts apache/hudi#3769
The original PR apache/hudi#3769 causes the `java.io.FileNotFoundException`
for the partition path upon the first commit to the MOR table with Hudi
kafka-connect sink using Java client.
```
[2021-11-03 11:27:48,702] ERROR [hudi-sink|task-0]
WorkerSinkTask{id=hudi-sink-0} RetriableException from SinkTask:
(org.apache.kafka.connect.runtime.WorkerSinkTask:600)
org.apache.kafka.connect.errors.RetriableException: Intermittent write
errors for Hudi for the topic/partition: hudi-test-topic:0 , ensuring kafka
connect will retry
at org.apache.hudi.connect.HoodieSinkTask.put(HoodieSinkTask.java:117)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieIOException: Error writing
records and ending commit 20211103112647 for partition 0
at
org.apache.hudi.connect.transaction.ConnectTransactionParticipant.handleEndCommit(ConnectTransactionParticipant.java:198)
at
org.apache.hudi.connect.transaction.ConnectTransactionParticipant.processRecords(ConnectTransactionParticipant.java:124)
at org.apache.hudi.connect.HoodieSinkTask.put(HoodieSinkTask.java:114)
... 11 more
Caused by: java.io.IOException: org.apache.hudi.exception.HoodieIOException:
Write records failed
... 14 more
Caused by: org.apache.hudi.exception.HoodieIOException: Write records failed
at
org.apache.hudi.connect.writers.BufferedConnectWriter.flushRecords(BufferedConnectWriter.java:125)
at
org.apache.hudi.connect.writers.AbstractConnectWriter.close(AbstractConnectWriter.java:95)
at
org.apache.hudi.connect.transaction.ConnectTransactionParticipant.handleEndCommit(ConnectTransactionParticipant.java:177)
... 13 more
Caused by: java.io.IOException:
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit
time 20211103112647
... 16 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert
for commit time 20211103112647
at
org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.execute(JavaUpsertPreppedDeltaCommitActionExecutor.java:96)
at
org.apache.hudi.table.HoodieJavaMergeOnReadTable.upsertPrepped(HoodieJavaMergeOnReadTable.java:50)
at
org.apache.hudi.table.HoodieJavaMergeOnReadTable.upsertPrepped(HoodieJavaMergeOnReadTable.java:40)
at
org.apache.hudi.client.HoodieJavaWriteClient.upsertPreppedRecords(HoodieJavaWriteClient.java:120)
at
org.apache.hudi.connect.writers.BufferedConnectWriter.flushRecords(BufferedConnectWriter.java:112)
... 15 more
Caused by: org.apache.hudi.exception.HoodieIOException: Failed to list base
files in partition partition-0
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:300)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:281)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestFileSlice(AbstractTableFileSystemView.java:577)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:101)
at
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestFileSlice(PriorityBasedFileSystemView.java:252)
at
org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:135)
at
org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:349)
at
org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.lambda$execute$0(JavaUpsertPreppedDeltaCommitActionExecutor.java:83)
at java.util.HashMap.forEach(HashMap.java:1289)
at
org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.execute(JavaUpsertPreppedDeltaCommitActionExecutor.java:80)
... 19 more
Caused by: java.io.FileNotFoundException: File
file:/tmp/hoodie/hudi-test-topic/partition-0 does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:674)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
at
org.apache.hudi.common.fs.FSUtils.getAllDataFilesInPartition(FSUtils.java:453)
at
org.apache.hudi.metadata.FileSystemBackedTableMetadata.getAllFilesInPartition(FileSystemBackedTableMetadata.java:62)
at
org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:129)
at
org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at
org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:290)
... 29 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
