Re: Tombstoned error and then OOM

2016-10-06 Thread kurt Greaves 
you'll still need to query all the data even if it's secondary indexed.

On 4 October 2016 at 17:13, INDRANIL BASU <indranil...@yahoo.com> wrote:

> The query has a where clause on a column which is a secondary index in the
> column family.
> E.g
> select * from test_schema.test_cf where status = 0;
> Here the status is integer column which is indexed.
>
> -- IB
>
> --
> *From:* kurt Greaves <k...@instaclustr.com>
> *To:* user@cassandra.apache.org; INDRANIL BASU <indranil...@yahoo.com>
> *Sent:* Tuesday, 4 October 2016 10:38 PM
> *Subject:* Re: Tombstoned error and then OOM
>
> This sounds like you're running a query that consumes a lot of memory. Are
> you by chance querying a very large partition or not bounding your query?
>
> I'd also recommend upgrading to 2.1.15, 2.1.0 is very old and has quite a
> few bugs.
>
> On 3 October 2016 at 17:08, INDRANIL BASU <indranil...@yahoo.com> wrote:
>
> Hello All,
>
>
>
> I am getting the below error repeatedly in the system log of C* 2.1.0
>
> WARN  [SharedPool-Worker-64] 2016-09-27 00:43:35,835
> SliceQueryFilter.java:236 - Read 0 live and 1923 tombstoned cells in
> test_schema.test_cf.test_cf_ col1_idx (see tombstone_warn_threshold). 5000
> columns was requested, slices=[-], delInfo={deletedAt=-
> 9223372036854775808, localDeletion=2147483647}
>
> After that NullPointer Exception and finally OOM
>
> ERROR [CompactionExecutor:6287] 2016-09-29 22:09:13,546
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 6287,1,main]
> java.lang. NullPointerException: null
> at org.apache.cassandra.service. CacheService$
> KeyCacheSerializer.serialize( CacheService.java:475)
> ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.service. CacheService$
> KeyCacheSerializer.serialize( CacheService.java:463)
> ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.cache. AutoSavingCache$Writer.
> saveCache(AutoSavingCache. java:225) ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at org.apache.cassandra.db. compaction.CompactionManager$
> 11.run(CompactionManager.java: 1061) ~[apache-cassandra-2.1.0.jar: 2.1.0]
> at java.util.concurrent. Executors$RunnableAdapter. call(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. FutureTask.run(Unknown Source)
> ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown
> Source) [na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown
> Source) [na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_80]
> ERROR [CompactionExecutor:9712] 2016-10-01 10:09:13,871
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 9712,1,main]
> java.lang. NullPointerException: null
> ERROR [CompactionExecutor:10070] 2016-10-01 14:09:14,154
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10070,1,main]
> java.lang. NullPointerException: null
> ERROR [CompactionExecutor:10413] 2016-10-01 18:09:14,265
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10413,1,main]
> java.lang. NullPointerException: null
> ERROR [MemtableFlushWriter:2396] 2016-10-01 20:28:27,425
> CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:
> 2396,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_80]
> at java.lang.Thread.start(Unknown Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.addWorker( Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.
> processWorkerExit(Unknown Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown
> Source) ~[na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_80]
>
> -- IB
>
>
>
>
>
>
>
>
> --
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
>
>


-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Re: Tombstoned error and then OOM

2016-10-04 Thread INDRANIL BASU 
The query has a where clause on a column which is a secondary index in the 
column family.E.g 
select * from test_schema.test_cf where status = 0; 
Here the status is integer column which is indexed. 
 -- IB

  From: kurt Greaves <k...@instaclustr.com>
 To: user@cassandra.apache.org; INDRANIL BASU <indranil...@yahoo.com> 
 Sent: Tuesday, 4 October 2016 10:38 PM
 Subject: Re: Tombstoned error and then OOM
   
This sounds like you're running a query that consumes a lot of memory. Are you 
by chance querying a very large partition or not bounding your query?

I'd also recommend upgrading to 2.1.15, 2.1.0 is very old and has quite a few 
bugs.

On 3 October 2016 at 17:08, INDRANIL BASU <indranil...@yahoo.com> wrote:

Hello All,



I am getting the below error repeatedly in the system log of C* 2.1.0

WARN  [SharedPool-Worker-64] 2016-09-27 00:43:35,835 SliceQueryFilter.java:236 
- Read 0 live and 1923 tombstoned cells in test_schema.test_cf.test_cf_ 
col1_idx (see tombstone_warn_threshold). 5000 columns was requested, 
slices=[-], delInfo={deletedAt=- 9223372036854775808, localDeletion=2147483647}
After that NullPointer Exception and finally OOM
ERROR [CompactionExecutor:6287] 2016-09-29 22:09:13,546 
CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor: 
6287,1,main]
java.lang. NullPointerException: null
    at org.apache.cassandra.service. CacheService$ 
KeyCacheSerializer.serialize( CacheService.java:475) 
~[apache-cassandra-2.1.0.jar: 2.1.0]
    at org.apache.cassandra.service. CacheService$ 
KeyCacheSerializer.serialize( CacheService.java:463) 
~[apache-cassandra-2.1.0.jar: 2.1.0]
    at org.apache.cassandra.cache. AutoSavingCache$Writer. 
saveCache(AutoSavingCache. java:225) ~[apache-cassandra-2.1.0.jar: 2.1.0]
    at org.apache.cassandra.db. compaction.CompactionManager$ 
11.run(CompactionManager.java: 1061) ~[apache-cassandra-2.1.0.jar: 2.1.0]
    at java.util.concurrent. Executors$RunnableAdapter. call(Unknown 
Source) ~[na:1.7.0_80]
    at java.util.concurrent. FutureTask.run(Unknown Source) ~[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown Source) 
[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown Source) 
[na:1.7.0_80]
    at java.lang.Thread.run(Unknown Source) [na:1.7.0_80]
ERROR [CompactionExecutor:9712] 2016-10-01 10:09:13,871 
CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor: 
9712,1,main]
java.lang. NullPointerException: null
ERROR [CompactionExecutor:10070] 2016-10-01 14:09:14,154 
CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor: 
10070,1,main]
java.lang. NullPointerException: null
ERROR [CompactionExecutor:10413] 2016-10-01 18:09:14,265 
CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor: 
10413,1,main]
java.lang. NullPointerException: null
ERROR [MemtableFlushWriter:2396] 2016-10-01 20:28:27,425 
CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter: 
2396,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[na:1.7.0_80]
    at java.lang.Thread.start(Unknown Source) ~[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor.addWorker( Unknown Source) 
~[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor. processWorkerExit(Unknown 
Source) ~[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown Source) 
~[na:1.7.0_80]
    at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown Source) 
~[na:1.7.0_80]
    at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_80]
-- IB




   



-- 
Kurt greavesk...@instaclustr.comwww.instaclustr.com

Re: Tombstoned error and then OOM

2016-10-04 Thread kurt Greaves 
This sounds like you're running a query that consumes a lot of memory. Are
you by chance querying a very large partition or not bounding your query?

I'd also recommend upgrading to 2.1.15, 2.1.0 is very old and has quite a
few bugs.

On 3 October 2016 at 17:08, INDRANIL BASU  wrote:

> Hello All,
>
>
>
> I am getting the below error repeatedly in the system log of C* 2.1.0
>
> WARN  [SharedPool-Worker-64] 2016-09-27 00:43:35,835
> SliceQueryFilter.java:236 - Read 0 live and 1923 tombstoned cells in
> test_schema.test_cf.test_cf_col1_idx (see tombstone_warn_threshold). 5000
> columns was requested, slices=[-], delInfo={deletedAt=-9223372036854775808,
> localDeletion=2147483647}
>
> After that NullPointer Exception and finally OOM
>
> ERROR [CompactionExecutor:6287] 2016-09-29 22:09:13,546
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 6287,1,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.service.CacheService$
> KeyCacheSerializer.serialize(CacheService.java:475)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.service.CacheService$
> KeyCacheSerializer.serialize(CacheService.java:463)
> ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.cache.AutoSavingCache$Writer.
> saveCache(AutoSavingCache.java:225) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at org.apache.cassandra.db.compaction.CompactionManager$
> 11.run(CompactionManager.java:1061) ~[apache-cassandra-2.1.0.jar:2.1.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent.FutureTask.run(Unknown Source)
> ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) [na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_80]
> ERROR [CompactionExecutor:9712] 2016-10-01 10:09:13,871
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 9712,1,main]
> java.lang.NullPointerException: null
> ERROR [CompactionExecutor:10070] 2016-10-01 14:09:14,154
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10070,1,main]
> java.lang.NullPointerException: null
> ERROR [CompactionExecutor:10413] 2016-10-01 18:09:14,265
> CassandraDaemon.java:166 - Exception in thread Thread[CompactionExecutor:
> 10413,1,main]
> java.lang.NullPointerException: null
> ERROR [MemtableFlushWriter:2396] 2016-10-01 20:28:27,425
> CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:
> 2396,5,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_80]
> at java.lang.Thread.start(Unknown Source) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) ~[na:1.7.0_80]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source) ~[na:1.7.0_80]
> at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_80]
>
> -- IB
>
>
>
>
>
>


-- 
Kurt Greaves
k...@instaclustr.com
www.instaclustr.com

Tombstoned error and then OOM

2016-10-03 Thread INDRANIL BASU 
Hello All,



I am getting the below error repeatedly in the system log of C* 2.1.0

WARN  [SharedPool-Worker-64] 2016-09-27 00:43:35,835 SliceQueryFilter.java:236 
- Read 0 live and 1923 tombstoned cells in test_schema.test_cf.test_cf_col1_idx 
(see tombstone_warn_threshold). 5000 columns was requested, slices=[-], 
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
After that NullPointer Exception and finally OOM
ERROR [CompactionExecutor:6287] 2016-09-29 22:09:13,546 
CassandraDaemon.java:166 - Exception in thread 
Thread[CompactionExecutor:6287,1,main]
java.lang.NullPointerException: null
    at 
org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475)
 ~[apache-cassandra-2.1.0.jar:2.1.0]
    at 
org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463)
 ~[apache-cassandra-2.1.0.jar:2.1.0]
    at 
org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:225)
 ~[apache-cassandra-2.1.0.jar:2.1.0]
    at 
org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1061)
 ~[apache-cassandra-2.1.0.jar:2.1.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) 
~[na:1.7.0_80]
    at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
[na:1.7.0_80]
    at java.lang.Thread.run(Unknown Source) [na:1.7.0_80]
ERROR [CompactionExecutor:9712] 2016-10-01 10:09:13,871 
CassandraDaemon.java:166 - Exception in thread 
Thread[CompactionExecutor:9712,1,main]
java.lang.NullPointerException: null
ERROR [CompactionExecutor:10070] 2016-10-01 14:09:14,154 
CassandraDaemon.java:166 - Exception in thread 
Thread[CompactionExecutor:10070,1,main]
java.lang.NullPointerException: null
ERROR [CompactionExecutor:10413] 2016-10-01 18:09:14,265 
CassandraDaemon.java:166 - Exception in thread 
Thread[CompactionExecutor:10413,1,main]
java.lang.NullPointerException: null
ERROR [MemtableFlushWriter:2396] 2016-10-01 20:28:27,425 
CassandraDaemon.java:166 - Exception in thread 
Thread[MemtableFlushWriter:2396,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method) ~[na:1.7.0_80]
    at java.lang.Thread.start(Unknown Source) ~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) 
~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(Unknown 
Source) ~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
~[na:1.7.0_80]
    at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_80]
-- IB