rasmus93 opened a new issue #8156:
URL: https://github.com/apache/pinot/issues/8156
Hi, seems like there is problem with running many queries in parallel in
latest build.
While performing load test I found such exceptions in logs:
```
2022/02/07 19:55:51.317 ERROR [InstanceRequestHandler]
[nioEventLoopGroup-3-1] Query processing error:
org.apache.thrift.protocol.TProtocolException: don't know what type: 15
at
org.apache.thrift.protocol.TCompactProtocol.getTType(TCompactProtocol.java:898)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.protocol.TCompactProtocol.readFieldBegin(TCompactProtocol.java:562)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:108)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest$InstanceRequestStandardScheme.read(InstanceRequest.java:679)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest$InstanceRequestStandardScheme.read(InstanceRequest.java:662)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest.read(InstanceRequest.java:569)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.core.transport.InstanceRequestHandler.channelRead0(InstanceRequestHandler.java:110)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.core.transport.InstanceRequestHandler.channelRead0(InstanceRequestHandler.java:56)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
and
```
2022/02/07 20:14:39.500 ERROR [InstanceRequestHandler]
[nioEventLoopGroup-3-1] Query processing error:
org.apache.thrift.protocol.TProtocolException: Required field 'query' is
unset! Struct:InstanceRequest(requestId:1635638400000, query:null)
at
org.apache.pinot.common.request.InstanceRequest.validate(InstanceRequest.java:629)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest$InstanceRequestStandardScheme.read(InstanceRequest.java:731)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest$InstanceRequestStandardScheme.read(InstanceRequest.java:662)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.common.request.InstanceRequest.read(InstanceRequest.java:569)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:81)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.thrift.TDeserializer.deserialize(TDeserializer.java:67)
~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.core.transport.InstanceRequestHandler.channelRead0(InstanceRequestHandler.java:110)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
org.apache.pinot.core.transport.InstanceRequestHandler.channelRead0(InstanceRequestHandler.java:56)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at
shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-3c98a4469644fe94c40ec004c0792870ef9d8af3]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
Such exception I get even if run load only with queries like `select
count(column) from test_table`. I can say that this exception is happened only
on high load, i.e. I found request which were failed in logs and tried to run
it separately then everything is ok.
Also I found that such problem exists in build with this commit -
[3c98a44696](https://github.com/apache/pinot/commit/3c98a4469644fe94c40ec004c0792870ef9d8af3),
but in build with commit -
[34f371d1cb](https://github.com/apache/pinot/commit/34f371d1cb6dbe9e174094d373fc8682e814ffdd)
everything is ok, so it looks like regression bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
