Abacn commented on issue #33753:
URL: https://github.com/apache/beam/issues/33753#issuecomment-5295501271
The load has been decreased multiple times since this this issue open. Did a
survey and we can revert some of the decreased load, but not all.
In particular Go GBK test 10b and 100b would still fail on OutOfMemoryError:
```
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.beam.sdk.values.KV.of(KV.java:45)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner$1.lambda$create$0(ExecutableStageDoFnOperator.java:965)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner$1$$Lambda/0x00007f97dfd5dde0.accept(Unknown
Source)
at
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(BeamFnDataInboundObserver.java:232)
at
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:186)
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor$ActiveBundle.close(SdkHarnessClient.java:565)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory$1.close(DefaultJobBundleFactory.java:562)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:1067)
at
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:100)
at
org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:133)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:1044)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator.flushData(ExecutableStageDoFnOperator.java:719)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.finish(DoFnOperator.java:691)
at
org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper$$Lambda/0x00007f97dfd2e248.run(Unknown
Source)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
```
Likely we need to use rockdb state backend instead of the default memory one
for Flink runner. This option hasn't exposed in Go SDK. And if does, not sure
it's working correctly as there isn't test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
