Re: endless full gc on one node

2016-01-17 Thread Kai Wang 
DuyHai,

In this case I didn't use batch, just bind a single PreparedStatement and
execute. Nor did I see any warning/error about batch being too large in the
log.

Thanks.

On Sat, Jan 16, 2016 at 6:27 PM, DuyHai Doan  wrote:

> "As soon as inserting started, one node started non-stop full GC. The
> other two nodes were totally fine"
>
> Just a guest, how did you insert data ? Did you use Batch statements ?
>
> On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang  wrote:
>
>> Hi,
>>
>> Recently I saw some strange behavior on one of the nodes of a 3-node
>> cluster. A while ago I created a table and put some data (about 150M) in it
>> for testing. A few days ago I started to import full data into that table
>> using normal cql INSERT statements. As soon as inserting started, one node
>> started non-stop full GC. The other two nodes were totally fine. I stopped
>> the inserting process, restarted C* on all the nodes. All nodes are fine.
>> But once I started inserting again, full GC kicked in on that node within a
>> minute.The insertion speed is moderate. Again, the other two nodes were
>> fine. I tried this process a couple of times. Every time the same node
>> jumped into full GC. I even rebooted all the boxes. I checked system.log
>> but found no errors or warnings before full GC started.
>>
>> Finally I deleted and recreated the table. All of sudden the problem went
>> away. The only thing I can think of is that table was created using STCS.
>> After I inserted 150M data into it, I switched it to LCS. Then I ran
>> incremental repair a couple of times. I saw validation and normal
>> compaction on that table as expected. When I recreated the table, I created
>> it with LCS.
>>
>> I don't have the problem any more but just want to share the experience.
>> Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
>> 7 and Java 8. All boxes have the identical configurations.
>>
>> Thanks.
>>
>
>

Re:Re: endless full gc on one node

2016-01-17 Thread xutom 
Hi Kai Wang,
I also encounter such issue a few days ago. I have 6 nodes, and I found 2 
nodes do endless full gc when I export ALL datas from C* using "Select * from 
table". I remove all datas of the 2 nodes and install Cassandra again, and the 
problem gone away.



At 2016-01-18 06:18:46, "Kai Wang"  wrote:

DuyHai,


In this case I didn't use batch, just bind a single PreparedStatement and 
execute. Nor did I see any warning/error about batch being too large in the log.


Thanks.



On Sat, Jan 16, 2016 at 6:27 PM, DuyHai Doan  wrote:

"As soon as inserting started, one node started non-stop full GC. The other two 
nodes were totally fine"


Just a guest, how did you insert data ? Did you use Batch statements ?


On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang  wrote:

Hi,


Recently I saw some strange behavior on one of the nodes of a 3-node cluster. A 
while ago I created a table and put some data (about 150M) in it for testing. A 
few days ago I started to import full data into that table using normal cql 
INSERT statements. As soon as inserting started, one node started non-stop full 
GC. The other two nodes were totally fine. I stopped the inserting process, 
restarted C* on all the nodes. All nodes are fine. But once I started inserting 
again, full GC kicked in on that node within a minute.The insertion speed is 
moderate. Again, the other two nodes were fine. I tried this process a couple 
of times. Every time the same node jumped into full GC. I even rebooted all the 
boxes. I checked system.log but found no errors or warnings before full GC 
started.


Finally I deleted and recreated the table. All of sudden the problem went away. 
The only thing I can think of is that table was created using STCS. After I 
inserted 150M data into it, I switched it to LCS. Then I ran incremental repair 
a couple of times. I saw validation and normal compaction on that table as 
expected. When I recreated the table, I created it with LCS.


I don't have the problem any more but just want to share the experience. Maybe 
someone has an theory on this? BTW I am running C* 2.2.4 with CentOS 7 and Java 
8. All boxes have the identical configurations.



Thanks.

endless full gc on one node

2016-01-16 Thread Kai Wang 
Hi,

Recently I saw some strange behavior on one of the nodes of a 3-node
cluster. A while ago I created a table and put some data (about 150M) in it
for testing. A few days ago I started to import full data into that table
using normal cql INSERT statements. As soon as inserting started, one node
started non-stop full GC. The other two nodes were totally fine. I stopped
the inserting process, restarted C* on all the nodes. All nodes are fine.
But once I started inserting again, full GC kicked in on that node within a
minute.The insertion speed is moderate. Again, the other two nodes were
fine. I tried this process a couple of times. Every time the same node
jumped into full GC. I even rebooted all the boxes. I checked system.log
but found no errors or warnings before full GC started.

Finally I deleted and recreated the table. All of sudden the problem went
away. The only thing I can think of is that table was created using STCS.
After I inserted 150M data into it, I switched it to LCS. Then I ran
incremental repair a couple of times. I saw validation and normal
compaction on that table as expected. When I recreated the table, I created
it with LCS.

I don't have the problem any more but just want to share the experience.
Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
7 and Java 8. All boxes have the identical configurations.

Thanks.

Re: endless full gc on one node

2016-01-16 Thread DuyHai Doan 
"As soon as inserting started, one node started non-stop full GC. The other
two nodes were totally fine"

Just a guest, how did you insert data ? Did you use Batch statements ?

On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang  wrote:

> Hi,
>
> Recently I saw some strange behavior on one of the nodes of a 3-node
> cluster. A while ago I created a table and put some data (about 150M) in it
> for testing. A few days ago I started to import full data into that table
> using normal cql INSERT statements. As soon as inserting started, one node
> started non-stop full GC. The other two nodes were totally fine. I stopped
> the inserting process, restarted C* on all the nodes. All nodes are fine.
> But once I started inserting again, full GC kicked in on that node within a
> minute.The insertion speed is moderate. Again, the other two nodes were
> fine. I tried this process a couple of times. Every time the same node
> jumped into full GC. I even rebooted all the boxes. I checked system.log
> but found no errors or warnings before full GC started.
>
> Finally I deleted and recreated the table. All of sudden the problem went
> away. The only thing I can think of is that table was created using STCS.
> After I inserted 150M data into it, I switched it to LCS. Then I ran
> incremental repair a couple of times. I saw validation and normal
> compaction on that table as expected. When I recreated the table, I created
> it with LCS.
>
> I don't have the problem any more but just want to share the experience.
> Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
> 7 and Java 8. All boxes have the identical configurations.
>
> Thanks.
>