Hi Kai Wang,
I also encounter such issue a few days ago. I have 6 nodes, and I found 2
nodes do endless full gc when I export ALL datas from C* using "Select * from
table". I remove all datas of the 2 nodes and install Cassandra again, and the
problem gone away.
At 2016-01-18 06:18:46, "Kai Wang" <dep...@gmail.com> wrote:
DuyHai,
In this case I didn't use batch, just bind a single PreparedStatement and
execute. Nor did I see any warning/error about batch being too large in the log.
Thanks.
On Sat, Jan 16, 2016 at 6:27 PM, DuyHai Doan <doanduy...@gmail.com> wrote:
"As soon as inserting started, one node started non-stop full GC. The other two
nodes were totally fine"
Just a guest, how did you insert data ? Did you use Batch statements ?
On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang <dep...@gmail.com> wrote:
Hi,
Recently I saw some strange behavior on one of the nodes of a 3-node cluster. A
while ago I created a table and put some data (about 150M) in it for testing. A
few days ago I started to import full data into that table using normal cql
INSERT statements. As soon as inserting started, one node started non-stop full
GC. The other two nodes were totally fine. I stopped the inserting process,
restarted C* on all the nodes. All nodes are fine. But once I started inserting
again, full GC kicked in on that node within a minute.The insertion speed is
moderate. Again, the other two nodes were fine. I tried this process a couple
of times. Every time the same node jumped into full GC. I even rebooted all the
boxes. I checked system.log but found no errors or warnings before full GC
started.
Finally I deleted and recreated the table. All of sudden the problem went away.
The only thing I can think of is that table was created using STCS. After I
inserted 150M data into it, I switched it to LCS. Then I ran incremental repair
a couple of times. I saw validation and normal compaction on that table as
expected. When I recreated the table, I created it with LCS.
I don't have the problem any more but just want to share the experience. Maybe
someone has an theory on this? BTW I am running C* 2.2.4 with CentOS 7 and Java
8. All boxes have the identical configurations.
Thanks.
