[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713023#comment-17713023
]
Stefan Miklosovic commented on CASSANDRA-18336:
---
what do you think about this? https://github.com/apache/cassandra/pull/2276
[~naizhen] [~maxwellguo] [~brandon.williams] [~yukim]
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712193#comment-17712193
]
maxwellguo commented on CASSANDRA-18336:
It seems the dir data have been remove when using mmap and the memmory is
limited
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711823#comment-17711823
]
Yuki Morishita commented on CASSANDRA-18336:
The `best_effort` behavior has been the same since 1.2.1. (CASSANDRA-4847)
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711763#comment-17711763
]
maxwellguo commented on CASSANDRA-18336:
of course~~~,and this behavior is also incorrect for databases. and I think
this [~yukim] may do some help too.
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 -
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711744#comment-17711744
]
Stefan Miklosovic commented on CASSANDRA-18336:
---
I do not think we should remove any sstables when OutOfMemoryError is thrown.
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log,
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711414#comment-17711414
]
maxwellguo commented on CASSANDRA-18336:
this is right and will mark the dir unread and unwriteable ，I am trying to find
way to distingush betweent out of memory and real disk failure causes. In this
case all seems to be io exception.
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711391#comment-17711391
]
Stefan Miklosovic commented on CASSANDRA-18336:
---
Following the code, it will remove SStables in the directories which became
"unreadable". You may have more than one data directory of Cassandra which
might be on different disks so upon getting read errors from so and so disk or
if data are somehow corrupted, it will remove them but other data in other
directories might remain untouched. I do not know if we want to do something
about it.
[~blerer] do you have any idea about the second part of this case? Can you shed
more light on the logic here? Is this something expected?
https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/service/DefaultFSErrorHandler.java#L69-L90
All after your "if".
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705277#comment-17705277
]
maxwellguo commented on CASSANDRA-18336:
Let me check whether it is a bug or not
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705242#comment-17705242
]
NAIZHEN QUE commented on CASSANDRA-18336:
-
if (e instanceof FSReadError)
{
File directory =
DisallowedDirectories.maybeMarkUnreadable(new File(e.path));
if (directory != null)
Keyspace.removeUnreadableSSTables(directory);
}
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705238#comment-17705238
]
NAIZHEN QUE commented on CASSANDRA-18336:
-
!4031679897782_.pic.jpg!
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705239#comment-17705239
]
NAIZHEN QUE commented on CASSANDRA-18336:
-
when the DiskFailurePolicy is best_effort. it will delete all sstables.
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Fix For: 4.0.x
>
> Attachments: 4031679897782_.pic.jpg, 4241679905694_.pic.jpg,
> system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701453#comment-17701453
]
NAIZHEN QUE commented on CASSANDRA-18336:
-
The version is 4.0.3 . All the data (sstables )on the node were deleted!
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Attachments: system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
> transaction log, deleting
>
[jira] [Commented] (CASSANDRA-18336) Sstables were cleared when restarting
[
https://issues.apache.org/jira/browse/CASSANDRA-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701229#comment-17701229
]
Brandon Williams commented on CASSANDRA-18336:
--
What version of C* is this against? Was any actual data lost? I tried this
against 4.0 and the behavior matched but all the data was still there so
nothing seems amiss.
> Sstables were cleared when restarting
> -
>
> Key: CASSANDRA-18336
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18336
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: NAIZHEN QUE
>Priority: Normal
> Attachments: system.log.2023-02-21.0
>
>
> 1.When this exception occurs in the system
> {code:java}
> //
> ERROR [CompactionExecutor:351627] 2023-02-21 17:59:20,721
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:351627,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:167)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
> at
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:365)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:337)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:172)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:124)
> at
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:64)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:137)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:193)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:298)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1016)
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:163)
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1013)
> {code}
> 2.Restart the node, Verifying logfile transaction ,All sstables are deleted
> {code:java}
> // code placeholder
> INFO [main] 2023-02-21 18:00:23,350 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Index.db
>
> INFO [main] 2023-02-21 18:00:23,615 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb-8819408-big-Data.db
>
> INFO [main] 2023-02-21 18:00:46,504 LogTransaction.java:240 - Unfinished
> transaction log, deleting
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b/nb_txn_compaction_c923b230-b077-11ed-a081-5d5a5c990823.log
>
> INFO [main] 2023-02-21 18:00:46,510 LogTransaction.java:536 - Verifying
> logfile transaction
> [nb_txn_compaction_461935b0-b1ce-11ed-a081-5d5a5c990823.log in
> /historyData/cassandra/data/kairosdb/data_points-870fab7087ba11eb8b50d3c6960df21b]
> INFO [main] 2023-02-21 18:00:46,517 LogTransaction.java:240 - Unfinished
>
