pvary commented on code in PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#discussion_r1430978486
##########
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##########
@@ -383,8 +381,10 @@ protected IcebergSource<RowData> source() {
.project(TestFixtures.TS_SCHEMA)
.splitSize(100L)
.streaming(true)
- .monitorInterval(Duration.ofMillis(2))
+ .monitorInterval(Duration.ofMillis(100))
Review Comment:
I run the tests during the whole day yesterday, and found that the previous
fix removed most of the flakiness, but we still have some issues which fails
around 1000-2000 runs.
In failure cases the records are emitted by the emitter, but none of the are
arriving to the Sink. I see the above error in the logs in this case:
`SocketTimeout` on the `sink coordinator`, and retries - this appears multiple
times before the timeout. Also the log is swamped with the planner logs.
With increasing the monitoring interval, I had 3500 successful runs in a
row, when I decided that I need to spend my time elsewhere. As you can imagine
this is nothing conclusive, and given the failure rate, nothing that could be
tracked easily, but seems like a good improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
