Marwen94 commented on issue #12950:
URL: https://github.com/apache/hudi/issues/12950#issuecomment-2718785664
Indeed @danny0405 , I see a lot of these :
```
{
"applicationARN": "xxxxxxx",
"applicationVersionId": "40",
"locationInformation":
"org.apache.hudi.common.table.view.PriorityBasedFileSystemView.handleRuntimeException(PriorityBasedFileSystemView.java:129)",
"logger":
"org.apache.hudi.common.table.view.PriorityBasedFileSystemView",
"message": "Got error running preferred function. Trying secondary",
"messageSchemaVersion": "1",
"messageType": "ERROR",
"threadName": "stream_write: default_database.hudi_access_logs (5/8)#0",
"throwableInformation":
"org.apache.hudi.exception.HoodieRemoteException: Connect to <some ip>:34135
[/<some ip>] failed: Connection timed out (Connection timed out)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:239)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFile(RemoteHoodieTableFileSystemView.java:304)\n\tat
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:100)\n\tat
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFile(PriorityBasedFileSystemView.java:157)\n\tat
org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:157)\n\tat
org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:126)\n\tat
org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:69)\n\tat
org.apache.hudi.io.FlinkWriteHandleFactory$CommitWriteHandleFact
ory.createMergeHandle(FlinkWriteHandleFactory.java:181)\n\tat
org.apache.hudi.io.FlinkWriteHandleFactory$BaseCommitWriteHandleFactory.create(FlinkWriteHandleFactory.java:123)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:459)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.access$000(HoodieFlinkWriteClient.java:77)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient$AutoCloseableWriteHandle.<init>(HoodieFlinkWriteClient.java:515)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient$AutoCloseableWriteHandle.<init>(HoodieFlinkWriteClient.java:507)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:148)\n\tat
org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:192)\n\tat
org.apache.hudi.sink.StreamWriteFunction.writeBucket(StreamWriteFunction.java:495)\n\tat
org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:467)
\n\tat
java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)\n\tat
org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:463)\n\tat
org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:137)\n\tat
org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:167)\n\tat
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)\n\tat
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)\n\tat
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:89)\n\tat
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:253)\n\tat
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandle
r.java:204)\n\tat
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:395)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192)\n\tat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:751)\n\tat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:361)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$18(StreamTask.java:1437)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:
50)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1425)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1382)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:488)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)\n\tat
org.apache.flink.streaming.ru
ntime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)\n\tat
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:122)\n\tat
org.apache.flink.streaming.runtime.
io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638)\n\tat
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)\n\tat
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat
org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat
java.base/java.lang.Thread.run(Thread.java:829)\nCaused by:
org.apache.hudi.org.apache.http.conn.HttpHostConnectException: Connect to <some
ip>:34135 [/<some ip>] failed: Connection timed out (Connection timed
out)\n\tat org.
apache.hudi.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)\n\tat
org.apache.hudi.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n\tat
org.apache.hudi.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)\n\tat
org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)\n\tat
org.apache.hudi.org
.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)\n\tat
org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n\tat
org.apache.hudi.org.apache.http.client.fluent.Request.execute(Request.java:151)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.get(RemoteHoodieTableFileSystemView.java:542)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:188)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:235)\n\t...
57 more\nCaused by: java.net.ConnectException: Connection timed out
(Connection timed out)\n\tat
java.base/java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)\n\tat
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Abst
ractPlainSocketImpl.java:255)\n\tat
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)\n\tat
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat
java.base/java.net.Socket.connect(Socket.java:609)\n\tat
org.apache.hudi.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)\n\tat
org.apache.hudi.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)\n\t...
71 more\n"
}
```
After which Hudi re-creates many In memory and remote views and seems to
retry the same thing again and again:
```
{
"applicationARN": "xxxxxxxxx",
"applicationVersionId": "40",
"locationInformation":
"org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:167)",
"logger": "org.apache.hudi.common.table.view.FileSystemViewManager",
"message": "Creating InMemory based view for basePath
s3a://<s3_bucket_name>/performance/test23.",
"messageSchemaVersion": "1",
"messageType": "INFO",
"threadName": "stream_write: default_database.hudi_access_logs (6/8)#0"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
