[
https://issues.apache.org/jira/browse/SPARK-35237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340723#comment-17340723
]
Kilyén Attila Örs commented on SPARK-35237:
-------------------------------------------
Hi we saw this issue when the Prometheus was scraping all the ports, hence also
the ports where spark worker talking to the driver. When we managed to specify
exactly which port to scrape the error went away.
> In k8s, during running spark job, IllegalArgumentException(too large frame)
> is raised on spark driver. It seems to related to prometheus
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-35237
> URL: https://issues.apache.org/jira/browse/SPARK-35237
> Project: Spark
> Issue Type: Question
> Components: Kubernetes
> Affects Versions: 3.0.2
> Reporter: Keunhyun Oh
> Priority: Major
>
> In k8s, during running spark job, IllegalArgumentException(too large frame)
> is raised on spark driver like that
> {code:java}
> 2021-04-26 13:59:26,961 WARN server.TransportChannelHandler: Exception in
> connection from /${Prometheus pod Ip addr}:51346
> java.lang.IllegalArgumentException: Too large frame: 5135603447297303916
> at
> org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119)
> at
> org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)
> at
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:98)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source) {code}
>
> ${Prometheus pod Ip addr} means prometheus pod's cluster ip in k8s.
> How to resolve the warning?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
