dawidwys commented on pull request #32: URL: https://github.com/apache/flink-benchmarks/pull/32#issuecomment-929184496
I tried changing the debloating target with basically a similar outcome. Changing the threshold also doesn't seem to help and does not have any visible effect. After some debugging the buffer size just jumps back and forth a lot, irrespective of the setting. I added logs in `BufferDebloater` when announcing the new size of buffers. I observed that in the setup from this PR, the throughput varied quite a lot. I tried increasing the `BUFFER_DEBLOAT_PERIOD` which helped with the throughput measurement, but did not solve the issue with unstable results (and too high results). One thing I noticed and haven't understood and solved yet is that I see that the `totalNumberOfBuffersInUse` also varies a lot: ``` 14:50:44,461 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 1964. Throughput: 26195, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 4 14:50:44,462 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 1230. Throughput: 32804, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 8 14:50:44,763 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 303. Throughput: 25314, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 25 14:50:44,764 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 527. Throughput: 31636, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 18 14:50:44,765 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 312. Throughput: 29151, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 28 14:50:44,784 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 874. Throughput: 26238, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 9 14:50:44,785 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 786. Throughput: 26225, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 10 14:50:44,785 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 873. Throughput: 26212, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 9 14:50:44,788 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 1574. Throughput: 26246, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 5 14:50:45,064 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 1812. Throughput: 30215, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 5 14:50:45,067 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 2864. Throughput: 47740, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 5 14:50:45,086 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 263. Throughput: 26333, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 30 14:50:45,087 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 282. Throughput: 26368, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 28 14:50:45,087 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 256. Throughput: 26135, lastEstimatedTimeToConsumeBuffers: PT0.333S, totalNumber: 34 14:50:45,091 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 442. Throughput: 26572, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 18 14:50:45,366 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 295. Throughput: 28591, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 29 14:50:45,369 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 1031. Throughput: 44710, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 13 14:50:45,668 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 321. Throughput: 31108, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 29 14:50:45,691 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 600. Throughput: 26033, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 13 14:50:45,691 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 561. Throughput: 26181, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 14 14:50:45,692 ERROR org.apache.flink.streaming.runtime.tasks.bufferdebloat.BufferDebloater [] - Announcing new buffer size: 522. Throughput: 26139, lastEstimatedTimeToConsumeBuffers: PT0.299S, totalNumber: 15 ``` which causes the buffer size to jump back and forth. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
