Konstantin Orlov created IGNITE-23116:
-----------------------------------------
Summary: Improve exception message on insertion under load
Key: IGNITE-23116
URL: https://issues.apache.org/jira/browse/IGNITE-23116
Project: Ignite
Issue Type: Improvement
Reporter: Konstantin Orlov
Tried to upload 6kk rows to a table and got
{{org.apache.ignite.sql.SqlException: IGN-CMN-65535
TraceId:9761599e-8900-4911-9177-0a3e9bcf370b Failed to put value into index}}
exception. I used multiple writers to keep CPU loaded on the level of 85-90%,
cluster of a single node with disabled fsync. Table has 6 sorted indexes.
First problem, is message is not very informative. Down start trace there is
{{IgniteInternalException: Failed to allocate temporary buffer for checkpoint
(increase checkpointPageBufferSize configuration property): default}}, but it's
unclear what does `default` means until you checked the place where exception
is thrown (apparently it's a name of the storage profile). Second, the only
usage of `checkpointPageBufferSize` is in the message of exception, so what
should be changed and where is not clear.
Another problem here, is that stack trace contains `CorruptedTreeException`,
which make the problem to seem more severe that it is.
Here is a full stack trace of the exception:
{code}
org.apache.ignite.internal.storage.StorageException: Failed to put value into
index
at
org.apache.ignite.internal.storage.pagememory.index.sorted.PageMemorySortedIndexStorage.lambda$put$2(PageMemorySortedIndexStorage.java:125)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.index.sorted.PageMemorySortedIndexStorage.put(PageMemorySortedIndexStorage.java:113)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:180)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:659)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:170)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:312)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:227)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
[?:?]
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:184)
[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:716)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
[disruptor-4.0.0.jar:?]
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
[disruptor-4.0.0.jar:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
B+Tree is corrupted [groupId=10, pageIds=[563027262832838], groupName=10,
msg=Runtime failure on search row: SortedIndexRow [rowId=RowId [partitionId=18,
uuid=00000191-9ea8-9562-a4f4-8d72b365595c]]]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6670)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2119)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.index.sorted.PageMemorySortedIndexStorage.lambda$put$2(PageMemorySortedIndexStorage.java:121)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
... 28 more
Caused by: org.apache.ignite.internal.lang.IgniteInternalException: Failed to
allocate temporary buffer for checkpoint (increase checkpointPageBufferSize
configuration property): default
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:318)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4112)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4370)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2196)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2171)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2171)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2093)
~[ignite-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.index.sorted.PageMemorySortedIndexStorage.lambda$put$2(PageMemorySortedIndexStorage.java:121)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
... 28 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
