[
https://issues.apache.org/jira/browse/IGNITE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Sizov updated IGNITE-20821:
-----------------------------------
Description:
AI3, rev. 8ebd9e842a2aaec5e1ae7e3b5e1cb36a040dc565
Benchmark:
[https://github.com/gridgain/YCSB/blob/ycsb-2023.7/ignite3/src/main/java/site/ycsb/db/ignite3/IgniteClient.java]
Cluster setup: 1 server node, aipersist, 25 partitions, raft.fsync=false
h1. Steps
Run the mentioned key-value benchmark to preload data (10 million unique keys):
{noformat}
Command line: -db site.ycsb.db.ignite3.IgniteClient -load -P
/opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
recordcount=250000 -p warmupops=50000 -p dataintegrity=true -p
measurementtype=timeseries -p status.interval=1 -p hosts=192.168.1.63 -p
recordcount=10000000 -p operationcount=10000000 -s
{noformat}
Observe that all 10M keys have been loaded:
{noformat}
[INSERT], Return=OK, 10000000
{noformat}
Run the 100% get operations over preloaded data:
{noformat}
Command line: -db site.ycsb.db.ignite3.IgniteClient -t -P
/opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
operationcount=250000 -p recordcount=250000 -p warmupops=50000 -p
dataintegrity=true -p measurementtype=timeseries -p status.interval=1 -p
hosts=192.168.1.63 -p recordcount=10000000 -p operationcount=10000000 -s
{noformat}
h1. Expected result
All 10M keys were read without errors
h1. Actual result
Reading 4341 keys out of 10M keys failed with exception like the following one:
{noformat}
Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 Row version lookup failed:
[rowId=RowId [partitionId=2, uuid=c18be8d8-e29d-4be9-9238-e85a8a18a3b9],
tableId=5, partitionId=2]
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:879)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.lambda$read$8(AbstractPageMemoryMvPartitionStorage.java:332)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:749)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.read(AbstractPageMemoryMvPartitionStorage.java:324)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.resolveRowByPkForReadOnly(PartitionReplicaListener.java:1843)
... 43 more
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
IGN-CMN-65535 TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 B+Tree is corrupted
[groupId=5, pageIds=[562958543400296], groupName=5, msg=Runtime failure on
lookup row:
org.apache.ignite.internal.storage.pagememory.mv.VersionChainKey@1b8df27b]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6724)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1580)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:870)
... 47 more
Caused by: java.lang.AssertionError
at
org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6464)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Search.run(BplusTree.java:369)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6451)
at
org.apache.ignite.internal.pagememory.util.PageHandler.readPage(PageHandler.java:157)
at
org.apache.ignite.internal.pagememory.datastructure.DataStructure.read(DataStructure.java:417)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.read(BplusTree.java:6693)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1654)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.doFind(BplusTree.java:1632)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1572)
... 48 more
{noformat}
was:
AI3, rev. 8ebd9e842a2aaec5e1ae7e3b5e1cb36a040dc565
Benchmark:
[https://github.com/gridgain/YCSB/blob/ycsb-2023.7/ignite3/src/main/java/site/ycsb/db/ignite3/IgniteClient.java]
Cluster setup: 1 server node, aipersist, 25 partitions, raft.fsync=false
h1. Steps
Run the mentioned key-value benchmark to preload data (10 million unique keys):
{noformat}
Command line: -db site.ycsb.db.ignite3.IgniteClient -load -P
/opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
recordcount=250000 -p warmupops=50000 -p dataintegrity=true -p
measurementtype=timeseries -p status.interval=1 -p hosts=192.168.1.63 -p
recordcount=10000000 -p operationcount=10000000 -s
{noformat}
Observe that all 10M keys have been loaded:
{noformat}
[INSERT], Return=OK, 10000000
{noformat}
Run the 100% get operations over preloaded data:
{noformat}
Command line: -db site.ycsb.db.ignite3.IgniteClient -t -P
/opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
operationcount=250000 -p recordcount=250000 -p warmupops=50000 -p
dataintegrity=true -p measurementtype=timeseries -p status.interval=1 -p
hosts=192.168.1.63 -p recordcount=10000000 -p operationcount=10000000 -s
{noformat}
h1. Expected result
All 10M keys were read without errors
h1. Actual result
Reading 4341 keys out of 10M keys failed with exception like the following one:
{noformat}
[17:26:22][ERROR][Thread-2] Error reading key: user7449531487681554067
org.apache.ignite.lang.IgniteException: Unable to close cursor [tableId=5]
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
~[?:?]
at
org.apache.ignite.internal.util.ExceptionUtils$1.copy(ExceptionUtils.java:754)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.util.ExceptionUtils$ExceptionFactory.createCopy(ExceptionUtils.java:688)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:525)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.util.ExceptionUtils.wrap(ExceptionUtils.java:589)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at org.apache.ignite.internal.client.ClientUtils.sync(ClientUtils.java:50)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.table.ClientKeyValueBinaryView.get(ClientKeyValueBinaryView.java:69)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.table.ClientKeyValueBinaryView.get(ClientKeyValueBinaryView.java:47)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at site.ycsb.db.ignite3.IgniteClient.read(IgniteClient.java:56)
[ignite3-binding-2023.7.jar:?]
at site.ycsb.DBWrapper.read(DBWrapper.java:145) [core-2023.7.jar:?]
at
site.ycsb.workloads.CoreWorkload.doTransactionRead(CoreWorkload.java:746)
[core-2023.7.jar:?]
at site.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:666)
[core-2023.7.jar:?]
at site.ycsb.ClientThread.run(ClientThread.java:145) [core-2023.7.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.lang.IgniteException: Unable to close cursor
[tableId=5]
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
~[?:?]
at
org.apache.ignite.internal.util.ExceptionUtils$1.copy(ExceptionUtils.java:754)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.util.ExceptionUtils$ExceptionFactory.createCopy(ExceptionUtils.java:688)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:525)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.TcpClientChannel.readError(TcpClientChannel.java:474)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:423)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.TcpClientChannel.onMessage(TcpClientChannel.java:231)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.io.netty.NettyClientConnection.onMessage(NettyClientConnection.java:111)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelRead(NettyClientMessageHandler.java:33)
~[ignite-client-3.0.0-SNAPSHOT.jar:?]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[netty-common-4.1.87.Final.jar:4.1.87.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[netty-common-4.1.87.Final.jar:4.1.87.Final]
... 1 more
{noformat}
> Sporadic exception on KeyValueView#get
> --------------------------------------
>
> Key: IGNITE-20821
> URL: https://issues.apache.org/jira/browse/IGNITE-20821
> Project: Ignite
> Issue Type: Bug
> Components: cache, clients
> Reporter: Ivan Artiukhov
> Priority: Major
> Labels: ignite-3, ignite3_performance
> Attachments: run269-logs.zip
>
>
> AI3, rev. 8ebd9e842a2aaec5e1ae7e3b5e1cb36a040dc565
> Benchmark:
> [https://github.com/gridgain/YCSB/blob/ycsb-2023.7/ignite3/src/main/java/site/ycsb/db/ignite3/IgniteClient.java]
>
> Cluster setup: 1 server node, aipersist, 25 partitions, raft.fsync=false
> h1. Steps
> Run the mentioned key-value benchmark to preload data (10 million unique
> keys):
>
> {noformat}
> Command line: -db site.ycsb.db.ignite3.IgniteClient -load -P
> /opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
> recordcount=250000 -p warmupops=50000 -p dataintegrity=true -p
> measurementtype=timeseries -p status.interval=1 -p hosts=192.168.1.63 -p
> recordcount=10000000 -p operationcount=10000000 -s
> {noformat}
> Observe that all 10M keys have been loaded:
>
>
> {noformat}
> [INSERT], Return=OK, 10000000
> {noformat}
> Run the 100% get operations over preloaded data:
>
> {noformat}
> Command line: -db site.ycsb.db.ignite3.IgniteClient -t -P
> /opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p
> operationcount=250000 -p recordcount=250000 -p warmupops=50000 -p
> dataintegrity=true -p measurementtype=timeseries -p status.interval=1 -p
> hosts=192.168.1.63 -p recordcount=10000000 -p operationcount=10000000 -s
> {noformat}
> h1. Expected result
> All 10M keys were read without errors
> h1. Actual result
> Reading 4341 keys out of 10M keys failed with exception like the following
> one:
> {noformat}
> Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 Row version lookup failed:
> [rowId=RowId [partitionId=2, uuid=c18be8d8-e29d-4be9-9238-e85a8a18a3b9],
> tableId=5, partitionId=2]
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:879)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.lambda$read$8(AbstractPageMemoryMvPartitionStorage.java:332)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:749)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.read(AbstractPageMemoryMvPartitionStorage.java:324)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.resolveRowByPkForReadOnly(PartitionReplicaListener.java:1843)
> ... 43 more
> Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
> IGN-CMN-65535 TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 B+Tree is
> corrupted [groupId=5, pageIds=[562958543400296], groupName=5, msg=Runtime
> failure on lookup row:
> org.apache.ignite.internal.storage.pagememory.mv.VersionChainKey@1b8df27b]
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6724)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1580)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:870)
> ... 47 more
> Caused by: java.lang.AssertionError
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6464)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$Search.run(BplusTree.java:369)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6451)
> at
> org.apache.ignite.internal.pagememory.util.PageHandler.readPage(PageHandler.java:157)
> at
> org.apache.ignite.internal.pagememory.datastructure.DataStructure.read(DataStructure.java:417)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.read(BplusTree.java:6693)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1654)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.doFind(BplusTree.java:1632)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1572)
> ... 48 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
