Marwen94 commented on issue #12950:
URL: https://github.com/apache/hudi/issues/12950#issuecomment-2721121655
Please ignore this log:
```
{
"applicationARN": "xxxxxxxxx",
"applicationVersionId": "40",
"locationInformation":
"org.apache.hudi.common.table.view.FileSystemViewManager.createInMemoryFileSystemView(FileSystemViewManager.java:167)",
"logger": "org.apache.hudi.common.table.view.FileSystemViewManager",
"message": "Creating InMemory based view for basePath
s3a://<s3_bucket_name>/performance/test23.",
"messageSchemaVersion": "1",
"messageType": "INFO",
"threadName": "stream_write: default_database.hudi_access_logs (6/8)#0"
}
```
I shared it because that's what follows this error in case it was relevant
for you but it is not the issue. Here is the issue:
````
{
"applicationARN": "xxxxxxx",
"applicationVersionId": "40",
"locationInformation":
"org.apache.hudi.common.table.view.PriorityBasedFileSystemView.handleRuntimeException(PriorityBasedFileSystemView.java:129)",
"logger":
"org.apache.hudi.common.table.view.PriorityBasedFileSystemView",
"message": "Got error running preferred function. Trying secondary",
"messageSchemaVersion": "1",
"messageType": "ERROR",
"threadName": "stream_write: default_database.hudi_access_logs (5/8)#0",
"throwableInformation":
"org.apache.hudi.exception.HoodieRemoteException: Connect to <some private
ip>:34135 [/<some private ip>] failed: Connection timed out (Connection timed
out)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:239)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFile(RemoteHoodieTableFileSystemView.java:304)\n\tat
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:100)\n\tat
org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFile(PriorityBasedFileSystemView.java:157)\n\tat
org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:157)\n\tat
org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:126)\n\tat
org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:69)\n\tat
org.apache.hudi.io.FlinkWriteHandleFactory$CommitWriteHandleFactory.createMergeHandle(FlinkWriteHandleFactory.java:181)\n\tat
org.apache.hudi.io.FlinkWriteHandleFactory$BaseCommitWriteHandleFactory.create(FlinkWriteHandleFactory.java:123)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:459)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.access$000(HoodieFlinkWriteClient.java:77)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient$AutoCloseableWriteHandle.<init>(HoodieFlinkWriteClient.java:515)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient$AutoCloseableWriteHandle.<init>(HoodieFlinkWriteClient.java:507)\n\tat
org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:148)\n\tat
org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:192)\n\tat
org.apache.hudi.sink.StreamWriteFunction.writeBucket(StreamWriteFunction.java:495)\n\tat
org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:467)\n\tat
java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)\n\tat
org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:463)\n\tat
org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:137)\n\tat
org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:167)\n\tat
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)\n\tat
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)\n\tat
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:89)\n\tat
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:253)\n\tat
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:204)\n\tat
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:395)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213)\n\tat
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192)\n\tat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:751)\n\tat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:361)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$18(StreamTask.java:1437)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)\n\ta
t
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1425)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1382)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:488)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)\n\tat
org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)\n\tat
org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:122)\n\tat
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638)\n\tat
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973)\n\tat
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)\n\tat
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat
org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat
java.base/java.lang.Thread.run(Thread.java:829)\nCaused by:
org.apache.hudi.org.apache.http.conn.HttpHostConnectException: Connect to
<some ip>:34135 [/<some ip>] failed: Connection timed out (Connection timed
out)\n\tat
org.apache.hudi.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)\n\tat
org.apache.hudi.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)\n\tat
org.apache.hudi.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\n\tat
org.apache.hudi.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)\n\tat
org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)\n\tat
org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)\n\tat
org.apache.hudi.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)\n\tat
org.apache.hudi.org.apache.http.client.fluent.Request.execute(Request.java:151)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.get(RemoteHoodieTableFileSystemView.java:542)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:188)\n\tat
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTabl
eFileSystemView.java:235)\n\t... 57 more\nCaused by:
java.net.ConnectException: Connection timed out (Connection timed out)\n\tat
java.base/java.net.PlainSocketImpl.socketConnect(Native Method)\n\tat
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)\n\tat
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)\n\tat
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)\n\tat
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)\n\tat
java.base/java.net.Socket.connect(Socket.java:609)\n\tat
org.apache.hudi.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)\n\tat
org.apache.hudi.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)\n\t...
71 more\n"
}
````
And this seems to bubble from
`org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint`
Again, Hudi works as expected when running the Flink app while disabling the
checkpointing but this is not production ready..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
