[
https://issues.apache.org/jira/browse/FLINK-25670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480053#comment-17480053
]
Igal Shilman commented on FLINK-25670:
--------------------------------------
You can use the 'payload_max_bytes' endpoint property to set a higher value.
The default limit stands on 32MB.
per your example:
{code:yaml}
apiVersion: v1
kind: ConfigMap
metadata:
namespace: statefun
name: module-config
data:
module.yaml: |+
kind: io.statefun.endpoints.v2/http
spec:
functions: example/*
urlPathTemplate: http://functions.statefun.svc.cluster.local:8000/statefun
transport:
payload_max_bytes: 104857600
timeouts:
call: 2min
---
kind: io.statefun.kafka.v1/ingress
spec:
id: example/in
address: kafka.statefun.svc.cluster.local:9092
consumerGroupId: my-group-id
topics:
- topic: names
valueType: example/MyMessage
targets:
- example/hello
{code}
> StateFun: Unable to handle oversize HTTP message if state size is large
> -----------------------------------------------------------------------
>
> Key: FLINK-25670
> URL: https://issues.apache.org/jira/browse/FLINK-25670
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
> Affects Versions: statefun-3.1.1
> Reporter: Kyle
> Priority: Major
> Attachments: 00-module.yaml, functions.py
>
>
> Per requirement we need to handle state which is about 500MB large (72MB
> state allocated in commented code as attached). However the HTTP message
> limit disallows us to send back large state to StateFun cluster after saving
> state in Stateful Function.
> Another question is whether large data is allowed to send to Stateful
> Function from ingress.
>
> 2022-01-17 07:57:18,416 WARN
> org.apache.flink.statefun.flink.core.nettyclient.NettyRequest [] - Exception
> caught while trying to deliver a message: (attempt
> #10)ToFunctionRequestSummary(address=Address(example, hello, 5555),
> batchSize=1, totalSizeInBytes=80, numberOfStates=2)
> org.apache.flink.shaded.netty4.io.netty.handler.codec.TooLongFrameException:
> Response entity too large: DefaultHttpResponse(decodeResult: success,
> version: HTTP/1.1)
> HTTP/1.1 200 OK
> Content-Type: application/octet-stream
> Content-Length: 40579630
> Date: Mon, 17 Jan 2022 07:57:18 GMT
> Server: Python/3.9 aiohttp/3.8.1
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:276)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.http.HttpObjectAggregator.handleOversizedMessage(HttpObjectAggregator.java:87)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.invokeHandleOversizedMessage(MessageAggregator.java:404)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:254)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
> ~[statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> [statefun-flink-distribution.jar:3.1.1]
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [statefun-flink-distribution.jar:3.1.1]
> at java.lang.Thread.run(Unknown Source) [?:?]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
